RPi: Kubernetes error function stuck in ImageInspectError state

[Gallouche]

Hi, I trying to deploy a home made function, but every time I, the pod that run that function goes in ImageInspectError state...
When I deploy a test function like figlet, everything work, but when it come to something I write myself, it fails.
I think I'm maybe missing something in the functioning because I'm pretty new to K8s and Openfaas.

I'm running OpenFaas with Kubernetes on a 2 nodes Raspberry PI 3B+ cluster.

Expected Behaviour

Pod should run.

Current Behaviour

NAMESPACE     NAME                                 READY     STATUS              RESTARTS   AGE
kube-system   etcd-masternode                      1/1       Running             1          1d
kube-system   kube-apiserver-masternode            1/1       Running             1          1d
kube-system   kube-controller-manager-masternode   1/1       Running             1          1d
kube-system   kube-dns-7f9b64f644-x42sr            3/3       Running             3          1d
kube-system   kube-proxy-wrp6f                     1/1       Running             1          1d
kube-system   kube-proxy-x6pvq                     1/1       Running             1          1d
kube-system   kube-scheduler-masternode            1/1       Running             1          1d
kube-system   weave-net-4995q                      2/2       Running             3          1d
kube-system   weave-net-5g7pd                      2/2       Running             3          1d
openfaas-fn   figlet-7f556fcd87-wrtf4              1/1       Running             0          4h
openfaas-fn   testfaceraspi-7f6fcb5897-rs4cq       0/1       ImageInspectError   0          2h
openfaas      alertmanager-66b98dd4d4-kcsq4        1/1       Running             1          1d
openfaas      faas-netesd-5b5d6d5648-mqftl         1/1       Running             1          1d
openfaas      gateway-846f8b5686-724q8             1/1       Running             2          1d
openfaas      nats-86955fb749-7vsbm                1/1       Running             1          1d
openfaas      prometheus-6ffc57bb8f-fpk6r          1/1       Running             1          1d
openfaas      queue-worker-567bcf4d47-ngsgv        1/1       Running             2          1d

The testfaceraspi doesn't run.

Logs from the pod :

$ kubectl logs testfaceraspi-7f6fcb5897-rs4cq -n openfaas-fn
Error from server (BadRequest): container "testfaceraspi" in pod "testfaceraspi-7f6fcb5897-rs4cq" is waiting to start: ImageInspectError

Pod describe :

$ kubectl describe pod -n openfaas-fn  testfaceraspi-7f6fcb5897-rs4cq 
Name:           testfaceraspi-7f6fcb5897-rs4cq
Namespace:      openfaas-fn
Node:           workernode/10.192.79.198
Start Time:     Thu, 12 Jul 2018 11:39:05 +0200
Labels:         faas_function=testfaceraspi
                pod-template-hash=3929761453
Annotations:    prometheus.io.scrape=false
Status:         Pending
IP:             10.40.0.16
Controlled By:  ReplicaSet/testfaceraspi-7f6fcb5897
Containers:
  testfaceraspi:
    Container ID:   
    Image:          gallouche/testfaceraspi
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImageInspectError
    Ready:          False
    Restart Count:  0
    Liveness:       exec [cat /tmp/.lock] delay=3s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [cat /tmp/.lock] delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      fprocess:  python3 index.py
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5qhnn (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-5qhnn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5qhnn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From                 Message
  ----     ------            ----                ----                 -------
  Warning  DNSConfigForming  2m (x1019 over 3h)  kubelet, workernode  Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch

And the event logs :

$ kubectl get events --sort-by=.metadata.creationTimestamp -n openfaas-fn
LAST SEEN   FIRST SEEN   COUNT     NAME                                              KIND         SUBOBJECT                        TYPE      REASON                  SOURCE                  MESSAGE
14m         1h           347       testfaceraspi-7f6fcb5897-rs4cq.1540db41e89d4c52   Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
4m          1h           75        figlet-7f556fcd87-wrtf4.1540db421002b49e          Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
10m         10m          1         testfaceraspi-7f6fcb5897-d6z78.1540df9ed8b91865   Pod                                           Normal    Scheduled               default-scheduler       Successfully assigned testfaceraspi-7f6fcb5897-d6z78 to workernode
10m         10m          1         testfaceraspi-7f6fcb5897.1540df9ed6eee11f         ReplicaSet                                    Normal    SuccessfulCreate        replicaset-controller   Created pod: testfaceraspi-7f6fcb5897-d6z78
10m         10m          1         testfaceraspi-7f6fcb5897-d6z78.1540df9eef3ef504   Pod                                           Normal    SuccessfulMountVolume   kubelet, workernode     MountVolume.SetUp succeeded for volume "default-token-5qhnn" 
4m          10m          27        testfaceraspi-7f6fcb5897-d6z78.1540df9eef5445c0   Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
8m          9m           8         testfaceraspi-7f6fcb5897-d6z78.1540df9f670d0dad   Pod          spec.containers{testfaceraspi}   Warning   InspectFailed           kubelet, workernode     Failed to inspect image "gallouche/testfaceraspi": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
9m          9m           7         testfaceraspi-7f6fcb5897-d6z78.1540df9f670fcf3e   Pod          spec.containers{testfaceraspi}   Warning   Failed                  kubelet, workernode     Error: ImageInspectError

Steps to Reproduce (for bugs)

1. Deploy OpenFaas on a 2 node k8s cluster
2. Create function with faas new testfaceraspi --lang python3-armhf
3. Add the following code in the handler.py :

import json

def handle(req):
	
	jsonl = json.loads(req)

	return ("Found " + str(jsonl["nbFaces"]) + " faces in OpenFaas Function on raspi !")

4. Change gateway and image in the .yml

provider:
  name: faas
  gateway: http://127.0.0.1:31112

functions:
  testfaceraspi:
    lang: python3-armhf
    handler: ./testfaceraspi
    image: gallouche/testfaceraspi

5. Run faas build -f testfacepi.yml
6. Login in DockerHub with docker login
7. Run faas push -f testfacepi.yml
8. Run faas deploy -f testfacepi.yml

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):

Commit: 3995a8197f1df1ecdf524844477cffa04e4690ea
Version: 0.6.11

• Docker version ( Full output from: docker version ):

Client:
 Version:       18.04.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    3d479c0
 Built: Tue Apr 10 18:25:24 2018
 OS/Arch:       linux/arm
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.04.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   3d479c0
  Built:        Tue Apr 10 18:21:25 2018
  OS/Arch:      linux/arm
  Experimental: false

• Are you using Docker Swarm (FaaS-swarm ) or Kubernetes (FaaS-netes)?
  Using kubernetes.

• Operating System and version (e.g. Linux, Windows, MacOS):

Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.4 (stretch)
Release:        9.4
Codename:       stretch

Thanks by advance !

[ivanayov]

Hi @Gallouche,

Is this a copy-paster error or there is an indentation problem in the function code?

def handle(req):
         return "hello world !"

[ivanayov]

Derek add label: support

[Gallouche]

Hi ! @ivanayov
It's a copy paste error ! Sorry !

[Gallouche]

@ivanayov I corrected it ! And add the events logs :)

[alexellis]

Hi, I was able to run your function with docker run and it worked fine.

I would suggest this is a Kubernetes on ARM issue, not an OpenFaaS issue. Could you raise it with the Kubernetes team?

Can you also give:

kubectl version

Alex

[Gallouche]

Hi @alexellis ! Yes I did the same with GKE and it works fine too...
Ok will check with them !

My kubernetes version :

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

I followed your Gist to deploy every things, with the help of other people in the comment to get it working.

Did you still have a working Raspi cluster with K8s and OpenFaas working actually ?

Thanks a lot !

[Gallouche]

I've post my question on Stackoverflow, I will keep you update. Thanks !

[ivanayov]

@Gallouche Yes, it was tested 3 weeks ago with K8s 1.10.2 on RPi 3 B+ and python3-armhf functions for a demo.

In your case there can be some incompatibility with the image, but it's hard to guess.
I think Kubernetes community could give more guidance.

We'd appreciate if you write back to confirm what's the actual issue.

Ivana

[Gallouche]

@ivanayov and which docker version ? Cause someone answer me on Stack and told that he saw this error once with a docker/kubernetes versions problem...

Yes for sure !

But @ivanayov @alexellis I just noticed that in the pod's description, that say :

Environment:
      fprocess:  python3 index.py

Shouldn't be python3-armhf ?

Thanks.

[alexellis]

No, in fact I showed above that your image works in my RPi. StackOverflow I'd say is not the correct forum for troubleshooting Kubernetes on ARM - raise an issue with the Kubernetes project or contact them via their Slack.

[alexellis]

Derek close: problem with user's environment