faas-cli deployment always trying to pull image from remote registry instead of local docker image

[swaroop-iquanti]

Deploy onto minikube is not using locally build docker image

Expected Behaviour

I was trying out python based serverless functions following the steps specified in https://blog.alexellis.io/introducing-functions-as-a-service/. I was able to build docker image as mentioned in the tutorial. When trying to deploy the newly created function's docker image, I was getting "ImagePullBackOff" in pod status. According to the post, if no image registry is mentioned it should default to the local image. Deployment is always trying to pull remote registry image instead of local image.

Function pods should be created using the local image instead of trying to pull from docker registry.

Current Behaviour

Possible Solution

imagePullPolicy if not mentioned should have defaulted to Never instead of Always.

Steps to Reproduce (for bugs)

1. faas-cli new --lang python3 hello-python-faas
2. faas-cli build -f hello-python-faas.yml
3. faas-cli deploy -f hello-python-faas.yml --gateway $(minikube service gateway-external --url)
4. kubectl get pods (Pod status of "hello-python-faas" is shown as ImagePullBackOff)

List of events as described by "kubectl describe pod hello-python-faas"
Events:
Type Reason Age From Message

---

Normal Scheduled 24m default-scheduler Successfully assigned hello-python-faas-6d4759cf6b-rsm68 to minikube
Normal SuccessfulMountVolume 24m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-8skmz"
Normal Pulling 23m (x4 over 24m) kubelet, minikube pulling image "hello-python-faas"
Warning Failed 23m (x4 over 24m) kubelet, minikube Failed to pull image "hello-python-faas": rpc error: code = Unknown desc = Error response from daemon: pull access denied for hello-python-faas, repository does not exist or may require 'docker login'
Warning Failed 23m (x4 over 24m) kubelet, minikube Error: ErrImagePull
Normal BackOff 22m (x6 over 24m) kubelet, minikube Back-off pulling image "hello-python-faas"
Warning Failed 4m (x81 over 24m) kubelet, minikube Error: ImagePullBackOff

Context

I was trying to explore OpenFaaS for Python3 Batch process workload in my organization

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):
  Client:
  Version: 17.12.0-ce
  API version: 1.23
  Go version: go1.9.2
  Git commit: c97c6d6
  Built: Wed Dec 27 20:10:36 2017
  OS/Arch: linux/amd64

Server:
Engine:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:45:38 2017
OS/Arch: linux/amd64
Experimental: false

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?
  Kubernetes through FaaS-netes

• Operating System and version (e.g. Linux, Windows, MacOS):
  Ubuntu 14.04 with VirtualBox 5.2

• Link to your project or a code example to reproduce issue:

[alexellis]

minikube is a special case - you will probably have to push to a registry for that - either the Docker Hub or a local registry - see also the Minkube registry add-ons.

Here are two more up to date tutorials you could checkout instead:

• https://blog.alexellis.io/first-faas-python-function/
• https://blog.alexellis.io/quickstart-openfaas-cli/

I'll close the issue but please keep commenting if you need to.

For an experience that feels more integrated you may just want to abandon minikube and use kubeadm directly on your host instead: https://blog.alexellis.io/kubernetes-in-10-minutes/

[masonoise]

Just to add something here, I'm encountering this same problem using Kubernetes via Docker for Mac. Launched k8s, set up Helm and use Helm to install openfaas, and deployed a test function but it's failing to pull the image. It sort of makes sense, but having to push to a registry while developing functions locally is not going to be much fun. Changing imagePullPolicy to Never doesn't seem to work but I haven't yet figured out why not.

[elijahgagne]

In case this helps someone, here's how I set things up to use a local registry

# Run a local registry on port 5000
docker run -d -p 5000:5000 --restart always --name registry registry:2

# Test pushing the alpine image to the local registry
docker pull alpine
docker tag alpine localhost:5000/alpine
docker push localhost:5000/alpine

# View the repositories
http://localhost:5000/v2/_catalog

# View the images in a repository
http://localhost:5000/v2/alpine/tags/list


# Set an environment variable to point faas-cli
[Environment]::SetEnvironmentVariable("OPENFAAS_URL", "http://localhost:31112", "Process")

# Create a root folder for the functions
mkdir -p ~/Desktop/func
cd ~/Desktop/func

# Pull the templates
faas-cli template pull
faas-cli new --list

# Create a new function
faas-cli new --lang csharp hello-csharp --prefix="localhost:5000"

# Build, push, and deploy
faas-cli build -f ./hello-csharp.yml
faas-cli push -f ./hello-csharp.yml
faas-cli deploy -f ./hello-csharp.yml

# Test it
echo Hi Elijah | faas-cli invoke hello-csharp

[masonoise]

Thanks for that. I might set things up to run a local registry, but it really shouldn't be needed. I removed openfaas, changed the chart to set imagePullPolicy to Never, and reinstalled, but it didn't make any difference, which was surprising. Not sure what's up.

[alexellis]

There is a working imagePullPolicy available for faas-netes now which was completed via #186 #185 - the person who completed the PR tested the subsequent release and confirmed the behaviour was operating correctly.

[alexellis]

@ericstoekl / @milsonian please can you test this feature and verify there is no regression?

[masonoise]

It doesn't seem to work for me. Verified that I'm using chart 1.0.13 and functions/faas-netesd:0.5.0. First I tried doing a helm update with --set faasnetesd.imagePullPolicy=never with no change in behavior. Then I did a helm delete of openfaas, edited faasnetesd.yaml to set imagePullPolicy: Never, and did a fresh helm update to reinstall. Still no change in behavior. I verified that the image it's looking for is there via docker images so presumably if it tries to do a pull it should work. There's no prefix needed for a local pull I assume...

[alexellis]

I have verified the behaviour works with both None and IfNotPresent. I edited the plain YAML diles and deployed.

I also assume that you have a typo in your command --set faasnetesd.imagePullPolicy=never - that should be correctly cased as per the docs as Never.

Example policy from faas-netesd deploys on DfM:

alexellis-a02:tmp alexellis$ kubectl get  -n openfaas-fn deploy/tester1 -o yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-05-09T19:32:11Z
  generation: 1
  labels:
    faas_function: tester1
  name: tester1
  namespace: openfaas-fn
  resourceVersion: "3223"
  selfLink: /apis/extensions/v1beta1/namespaces/openfaas-fn/deployments/tester1
  uid: abb1be9a-53bf-11e8-aa8d-025000000001
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      faas_function: tester1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io.scrape: "false"
      creationTimestamp: null
      labels:
        faas_function: tester1
      name: tester1
    spec:
      containers:
      - env:
        - name: fprocess
          value: ./handler
        image: tester1:latest
        imagePullPolicy: Never
        livenessProbe:
          exec:
            command:
            - cat
            - /tmp/.lock
          failureThreshold: 3
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: tester1
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/.lock
          failureThreshold: 3
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-05-09T19:32:20Z
    lastUpdateTime: 2018-05-09T19:32:20Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[secana]

Same behavior on Kubernetes with Docker for Windows. It's not possible to run a local image since k8s always searches on hub.docker.

Failed to pull image "local-image": rpc error: code = Unknown desc = Error response from daemon: pull access denied for local-image, repository does not exist or may require 'docker login'

[alexellis]

Please open a new issue so that you've filled in the whole template. You will need to use the IfNotPresent pull policy for the local library.

[alexellis]

Derek lock