Mirror all k8s CSI container images used to docker.io

[mysticaltech]

Hello, @LKaemmerling, writing this issue as maintainer of kube-hetzner (with 15k downloads on Terraform cloud).

We have been experiencing lots of CSI container image pull 403s coming from both k8s.gcr.io and the new registry.k8s.io, and it turns out we are not the only ones.

After investigation by the Kubernetes team, they concluded that it was coming from the underlying gcr system, and more precisely an LB security middleware called by Google Cloud called Cloud Armor which has blacklisted far too many Hetzner Cloud IPs.

As you can imagine this is a huge problem for our users, and we are seeing many complaints and issues because of it. And the Kubernetes team admitted that they cannot do anything about it, and it is Google's fault and they do not have the resources to address the problem. Instead, they are recommending we create a mirror, which we really do not have the resources to do at this time.

So a quick solution, of course, is to advise users to use private container registries and proxies like jFrog or Gitlab, and we support that. But JUST TO GET THE CLUSTER BOOTED IN GOOD SHAPE, could you please mirror all registry.k8s.io CSI images to your docker.io? Because that pulls just fine on Hetzner.

It would be a huge help to us! As we deploy your CSI manifest as is, and cannot find the right tags for these images on docker.io currently, even if we were to dynamically replace the main link on the fly during deployment time.

Please do consider it, as a lot of Hetzner Cloud nodes get deployed through our project, and this small gesture would help us provide a far better experience out of the box.

[apricote]

Hey @mysticaltech, as I wrote in the thread on registry.k8s.io we are aware of sporadic issues where IP addresses are blocked by Google & AWS. The same issue happens with hub.docker.com, so mirroring the images there might not improve the situation, while adding a lot of work and infrastructure to our efforts.

It would help if you (and your users) could send these issues to our support and mention that they have issues pulling csi images from the registry along with the IP that they are pulling from.

[mysticaltech]

Thanks @apricote, I will pass on the info!