Pods stuck on ImagePullBackOff status

[kollender]

After ksync installation, tried to run ksync init and got:

Cluster Config                              ✓
Cluster Connection                          ✓
Cluster Version                             ✓
Cluster Permissions                         ✓

==== Cluster Environment ====
Adding ksync to the cluster                 ✓
Waiting for pods to be healthy              ✘
↳       Timed out waiting for the ksync daemonset to be healthy.

To debug, you can:
- Run 'kubectl --namespace=kube-system --context= get pods -lapp=ksync' to look at what's going on.
- Run 'ksync doctor' to do an in-depth check of your system and the cluster.
FATA[0070]

When I run ksync doctor I get:

Cluster Config                              ✓
Cluster Connection                          ✓
Cluster Version                             ✓
Cluster Permissions                         ✓
Cluster Service                             ✓
Service Health                              ✘
↳       Cluster service is not healthy.

- If you just ran init, wait a little longer and try again.
- Run 'kubectl --namespace=kube-system --context= get pods -lapp=ksync' to look at what's going on.

The output of ksync version:

        Version:    Release
        Go Version: go1.13.4
        Git Commit: a5628b9
        Git Tag:    refs/tags/0.4.1
        Built:      Wed Nov 13 18:42:02 +0000 2019
        OS/Arch:    darwin/amd64
FATA[0005] no healthy nodes found

All of the pods are in the same status (kubectl --namespace=kube-system --context= get pods -lapp=ksync):

NAME          READY   STATUS             RESTARTS   AGE
ksync-7zgft   0/2     ImagePullBackOff   0          24m

The image in the DS is ksync/ksync:git-a5628b9.

The K8s cluster is a GKE cluster, and this is the output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-gke.17", GitCommit:"188432a69210ca32cafded81b4dd1c063720cac0", GitTreeState:"clean", BuildDate:"2019-11-13T20:47:11Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

[grampelberg]

@timfallmk any ideas why the image hash doesn't exist?

[timfallmk]

I was afraid of this. It's probably because of my messing with release process. I'll push that tag again manually.

[timfallmk]

@grampelberg Looks like I was wrong, the image appears to exist: https://hub.docker.com/layers/ksync/ksync/git-a5628b9/images/sha256-fcb07b0634571432d76736b695c676365907b4215d2f0edd179a6ef2df2fa486

@kollender Can you try again and see if it's a transient error?

[kollender]

@timfallmk again :\

==== Preflight checks ====
Cluster Config                              ✓
Cluster Connection                          ✓
Cluster Version                             ✓
Cluster Permissions                         ✓

==== Cluster Environment ====
Adding ksync to the cluster                 ✓
Waiting for pods to be healthy              ✘
↳	Timed out waiting for the ksync daemonset to be healthy.

To debug, you can:
- Run 'kubectl --namespace=kube-system --context= get pods -lapp=ksync' to look at what's going on.
- Run 'ksync doctor' to do an in-depth check of your system and the cluster.
FATA[0076]

And the image in the deamonset is ksync/ksync:git-a5628b9

[grampelberg]

What's in the logs?

[kollender]

@grampelberg what is the command for that?
I've Just run ksync doctor and the output is the same as I wrote earlier.

[grampelberg]

kubectl logs

Honestly it sounds like you've got connectivity issues to docker hub. Can you run any other images like alpine?

[kollender]

Here are some logs:
k describe ds ksync -n kube-system :

Name:           ksync
Selector:       app=ksync,name=ksync
Node-Selector:  beta.kubernetes.io/os=linux
Labels:         app=ksync
                name=ksync
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 6
Current Number of Nodes Scheduled: 6
Number of Nodes Scheduled with Up-to-date Pods: 6
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 6 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=ksync
                name=ksync
  Annotations:  forceUpdate: 1575936373
  Containers:
   ksync:
    Image:      ksync/ksync:git-a5628b9
    Port:       40321/TCP
    Host Port:  0/TCP
    Command:
      /radar
      --log-level=debug
      serve
    Environment:
      RADAR_POD_NAME:   (v1:metadata.name)
    Mounts:
      /var/run/docker.sock from dockersock (rw)
   syncthing:
    Image:       ksync/ksync:git-a5628b9
    Ports:       8384/TCP, 22000/TCP
    Host Ports:  0/TCP, 0/TCP
    Command:
      /syncthing/syncthing
      -home
      /var/syncthing/config
      -gui-apikey
      ksync
      -verbose
    Liveness:     tcp-socket :8384 delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:    tcp-socket :8384 delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/lib/docker from dockerfs (rw)
      /var/lib/kubelet from kubelet (rw)
      /var/run/docker.sock from dockersock (rw)
  Volumes:
   dockerfs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker
    HostPathType:
   dockersock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:
   kubelet:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet
    HostPathType:
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-5c76n
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-thnlh
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-lxppk
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-8x8z9
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-vqjcj
  Normal  SuccessfulCreate  106s  daemonset-controller  Created pod: ksync-994bz

k get po -n kube-system -l app=ksync gives me back:

NAME          READY   STATUS             RESTARTS   AGE
ksync-5c76n   0/2     ErrImagePull       0          4m30s
ksync-8x8z9   0/2     ErrImagePull       0          4m30s
ksync-994bz   0/2     ErrImagePull       0          4m30s
ksync-lxppk   0/2     ImagePullBackOff   0          4m30s
ksync-thnlh   0/2     ImagePullBackOff   0          4m30s
ksync-vqjcj   0/2     ErrImagePull       0          4m30s

k logs -f ksync-5c76n -n kube-system -c ksync:

Error from server (BadRequest): container "ksync" in pod "ksync-5c76n" is waiting to start: trying and failing to pull image

[grampelberg]

Yup, looks like you've got issues pulling the image, not much we can do to help out with that I'm afraid =/

[kollender]

Yeah, seems like that.
I will close the ticket and try to find the issue on our side.
Thank you for your help guys!