Move to registry.k8s.io? Images are stuck on ImagePullBackOff

[melalj]

After using the deploy script in https://raw.githubusercontent.com/hetznercloud/csi-driver/main/deploy/kubernetes/hcloud-csi.yml

I get my pod hcloud-csi-node and hcloud-csi-controller stuck in ImagePullBackOff

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  5m12s                default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         3m8s                 default-scheduler  Successfully assigned kube-system/hcloud-csi-controller-bb7658b8f-5fbbq to mycluster-it05-worker-large-lmb
  Normal   Pulled            3m7s                 kubelet            Successfully pulled image "hetznercloud/hcloud-csi-driver:2.1.0" in 130.499337ms
  Warning  Failed            3m7s                 kubelet            Failed to pull image "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2": failed to resolve reference "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2": pulling from host k8s.gcr.io failed with status code [manifests v2.2.2]: 403 Forbidden
  Warning  Failed            3m7s                 kubelet            Error: ErrImagePull
  Normal   Pulling           3m7s                 kubelet            Pulling image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
  Warning  Failed            3m7s                 kubelet            Failed to pull image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0": failed to resolve reference "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0": pulling from host k8s.gcr.io failed with status code [manifests v1.2.0]: 403 Forbidden
  Warning  Failed            3m7s                 kubelet            Error: ErrImagePull
  Normal   Pulling           3m7s                 kubelet            Pulling image "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
  Normal   Created           3m7s                 kubelet            Created container hcloud-csi-driver
  Warning  Failed            3m7s                 kubelet            Error: ErrImagePull
  Normal   Pulling           3m7s                 kubelet            Pulling image "hetznercloud/hcloud-csi-driver:2.1.0"
  Normal   Pulling           3m7s                 kubelet            Pulling image "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"
  Warning  Failed            3m7s                 kubelet            Failed to pull image "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1": failed to resolve reference "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1": pulling from host k8s.gcr.io failed with status code [manifests v3.2.1]: 403 Forbidden
  Warning  Failed            3m7s                 kubelet            Failed to pull image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": failed to resolve reference "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": pulling from host k8s.gcr.io failed with status code [manifests v2.3.0]: 403 Forbidden
  Normal   Pulling           3m7s                 kubelet            Pulling image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0"
  Normal   Started           3m7s                 kubelet            Started container hcloud-csi-driver
  Warning  Failed            3m7s                 kubelet            Error: ErrImagePull
  Warning  Failed            3m6s                 kubelet            Error: ImagePullBackOff
  Warning  Failed            3m6s                 kubelet            Error: ImagePullBackOff
  Normal   BackOff           3m6s                 kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
  Warning  Failed            3m6s                 kubelet            Error: ImagePullBackOff
  Normal   BackOff           3m6s                 kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
  Normal   BackOff           3m6s                 kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0"
  Warning  Failed            3m6s                 kubelet            Error: ImagePullBackOff
  Normal   BackOff           3m5s (x2 over 3m6s)  kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"

Also it seems that k8s is moving toward a new registry: kubernetes/kubernetes#109938

Any help would be much appreciated!

[apricote]

In general I expect the old registry to keep working for a time, although new versions might not be pushed there. If you have any other issue reference for shutting down k8s.gcr.io, please send it to me.

I can confirm that pulling the images from k8s.gcr.io fails at the moment and works when pulling from registry.k8s.io. I think this is a temporary outage of k8s.gcr.io, but Google Cloud Status does not report anything.

You can try to replace all k8s.gcr.io/ references with registry.k8s.io/and deploy that. It should work, but I did not verify it.

[apricote]

Pulling images from k8s.gcr.io works again for me. Going to close this issue.

If you have any further problems or questions, please feel free to reopen the issue or to open a new one.

[mysticaltech]

Pulling images from k8s.gcr.io works again for me. Going to close this issue.

If you have any further problems or questions, please feel free to reopen the issue or to open a new one.

@apricote It works most of the time, but some Hetzner cloud IPs are still blacklisted somehow (we are seeing this a lot on https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner).

But they are working on the fix over at kubernetes/registry.k8s.io#138.

[mysticaltech]

@apricote FYI, it turns out that a good bunch of Hetzner IPs are blacklisted by gcr the underlying system of the above registries.

Today I discussed on the issue above with members of the Kube team, they said it comes from Cloud Armor, and that they cannot do anything. They asks as to deploy our own mirror.

Please, would it be possible to mirror all the CSI needed images to your hetznercloud dockerhuh account and use those in the manifest? This will allow everyone to bypass the blacklisting.

Our project kube-hetzner/terraform-hcloud-kube-hetzner have had 12.5k downloads and we're nearing +500 downloads per week and 800 stars. But this issue is really causing lots of problems.

We are too small to deploy our own mirror for CSI images. Please consider moving them to dockerhub! 🙏