How to fix CSI Image pull error on a single node

[pbnsilva]

Hi,

I've made a fresh install, where my terraform config is the kube.tf example with some extra nodes and longhorn enabled:

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cpx11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count = 0
    },
    {
      name        = "agent-xlarge",
      server_type = "cpx41",
      location    = "nbg1",
      labels = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count = 0
    },
    {
      name        = "agent-xxlarge",
      server_type = "cpx51",
      location    = "nbg1",
      labels = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count = 3
    }
  ]

After it is installed, the hcloud-csi-node pod on a single node keeps crashing due to:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  2m42s                  default-scheduler  Successfully assigned kube-system/hcloud-csi-node-swkbt to k3s-agent-xxlarge-iog
  Normal   Pulling    2m38s                  kubelet            Pulling image "hetznercloud/hcloud-csi-driver:2.1.0"
  Normal   Pulled     2m34s                  kubelet            Successfully pulled image "hetznercloud/hcloud-csi-driver:2.1.0" in 4.139931398s
  Normal   Created    2m33s                  kubelet            Created container hcloud-csi-driver
  Normal   Started    2m33s                  kubelet            Started container hcloud-csi-driver
  Warning  Failed     2m33s                  kubelet            Failed to pull image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": failed to resolve reference "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0": pulling from host k8s.gcr.io failed with status code [manifests v2.3.0]: 403 Forbidden
  Warning  Failed     2m33s                  kubelet            Error: ErrImagePull
  Normal   BackOff    2m31s (x2 over 2m32s)  kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
  Warning  Failed     2m31s (x2 over 2m32s)  kubelet            Error: ImagePullBackOff
  Warning  Failed     2m31s (x2 over 2m32s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m31s (x2 over 2m32s)  kubelet            Back-off pulling image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0"
  Warning  Unhealthy  2m18s (x3 over 2m22s)  kubelet            Liveness probe failed: Get "http://10.42.2.2:9808/healthz": dial tcp 10.42.2.2:9808: connect: connection refused
  Normal   Pulling    2m18s (x2 over 2m38s)  kubelet            Pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
  Warning  Failed     2m17s (x2 over 2m38s)  kubelet            Error: ErrImagePull
  Warning  Failed     2m17s (x2 over 2m38s)  kubelet            Failed to pull image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0": failed to resolve reference "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0": pulling from host k8s.gcr.io failed with status code [manifests v2.2.0]: 403 Forbidden

So, 403 Forbidden when pulling from the registry.

I've tried destroying and recreating, but there's always a single node where this happens.

Any tips?

Thank you

[mysticaltech]

This is an issue we have been seeing a few times, please search in the issues and discussions. It seems that github rate limits the container registry requests. It can be solved by using another container registry via the k3s_registries variable in kube.tf.

[mysticaltech]

This is a duplicate of #442, but not everyone can deploy his own private registry. @pbnsilva did it finally work for you? Please try again, as is, without doing anything. It might work now, as I think gcr.io was overwhelmed about two weeks ago, but now we are not hearing more about this issue. Please confirm if possible!

[mysticaltech]

@kube-hetzner/core This problem is still raging! I believe it is k8s.gcr.io that is rate-limiting / blacklisting some IPs on Hetzner.

It is mainly the CSI manifest that uses those, and since it passes through kustomize, we can easily alter those. But where do you think we can find those same containers and versions?

See the screenshot below with the 403 errors.

[mysticaltech]

Turns out that even with registry.k8s.io, this rate limit things gets it with the csi yaml, probably because it fetches a lot of images at the same time.

[mysticaltech]

Hence, I am now considering deploying a small lightweight container registry, trow for instance is a lot lighter than harbor and would do the job like a charm!

@kube-hetzner/core What do you think?

[mysticaltech]

Here's the main guide https://github.com/Extrality/trow/blob/main/docs/USER_GUIDE.md#proxying-other-registries-and-mutatingwebhook

And a much better helm values.yaml. I think we can work with this!

https://github.com/Extrality/trow/blob/main/charts/trow/values.yaml#L30

[mysticaltech]

Hey @valkenburg-prevue-ch, did you get a chance to have a look at the above? It's really phenomenal and answer all our requirement. If you can test it, it would be great, otherwise, I will give it a shot as soon as I get some free time.

Also FYI, they are discussing merging the above into the main project, and even if that does not happen, we use the latest fork that is maintained and actively used by the author. Trow-Registry/trow#329

Last, but not least, good news from Hetzner, they are requesting that we signal blacklisted IPs. kubernetes/registry.k8s.io#138 (comment)

[valkenburg-prevue-ch]

Hey @mysticaltech , I have not yet looked into Trow, but I am not so sure about deploying it inside the cluster. Some containers need to be pulled before Trow can come up, notably the storage related services. So how does the k3s configuration work then?

[mysticaltech]

@valkenburg-prevue-ch CSI stuff always run last! I see it every time I do a deployment. So not a problem at all, worst case it ImagePullBackoff until trow is ready.

[mysticaltech]

Or it just does what it does always, and if it turns out that the node is blackisted, it will start working when trow kicks in.

[mysticaltech]

After doing a bunch of testing, it's become clear that some Hetzner IPs are blacklisted.

[valkenburg-prevue-ch]

Ouch, bad news about the blacklisting. Assuming you mean hosting a container registry publicly, wouldn't that be possible with GitHub? Or do you mean as a part of applying terraform, in order to minimize the simultaneous downloads of the same image?

[mysticaltech]

[mysticaltech]

It also seems that pulling one image, automatically unlocks all of them, effectively unbanning the node, so the script above is overdoing it, just to be thorough and check that all the needed images are indeed available.

[captnCC]

@mysticaltech Have you any idea, how this is working? I kinda staggered that brute forcing to cancel rate limiting or blacklisting is working 😳

[mysticaltech]

@captnCC I swear... me too. My best guess would be that it's not actually rate-limiting, but just a problem that came with the phasing out of k8s.gcr.io and transition to registry.k8s.io. They actually warned that issues may arise!

Somehow, requesting multiple times in a row probably finds a server willing to serve the image and sticks with it. That is what I think is happening.

OR and that is even more likely, after a number of failed requests, the server gives a chance again to the IP to see if the requests stop.

All speculations, but it works and hopefully all of this is nothing but a temporary measure.

[mysticaltech]

[valkenburg-prevue-ch]

To answer that last question: I am not affected by this issue, not currently using this csi driver.

[mysticaltech]

@valkenburg-prevue-ch Out of curiosity, what are you using? Or you do not need volumes?

[valkenburg-prevue-ch]

Maybe I missed some point, but I am using these flags:
enable_longhorn = true
disable_hetzner_csi = true
And I use two longhorn storage classes, one for large storage on hetzner ssd's, one using node storage (with redundancy) for databases.

[valkenburg-prevue-ch]

P.S. to this comment from 2 weeks back: I do have the issue now, with the secrets-store-csi-driver which is also pulled from registry.k8s.io. Anyway, I understand we'll just have to sit tight...

[mysticaltech]

@valkenburg-prevue-ch Yes, but if you are willing to install a registry of your own, you can proxy the request automatically by using the k3s_registries variable, see kube.tf.example for more details.

Or, you can SSH into the registry and try the fix above which worked for me and others in the past.

Or try this hack, which basically consists in blocking the IPs as static to force Hetzner to allocate new ones on cluster creation. #524 (comment)

[aleksasiriski]

This happened to me and the fix.sh script didn't work.. I had to delete my cluster project, then create new one, buy 10 IPs so that I got all of the old ones, create a new cluster which FINALLY got all new IPs and only after that delete the reserved IPs that were blacklisted...

[aleksasiriski]

[mysticaltech]

@aleksasiriski k8s.gcr.io is moving to registry.k8s.io, so I am hoping that these issues are temporary.

[valkenburg-prevue-ch]

Well, one more option is to manually reassign a new ip to the malfunctioning node, in console.hetzner.cloud. Worked for me, as only a single node had an ip which was blocked by many services. I restarted the node, but not sure if that was necessary?

Makes you wonder what kind of security issues some Hetzner users must have had, probably had a machine compromised and being used in a ddos attack. Hope it wasn't me. 🫣

[aleksasiriski]

I tried that but the cluster couldn't read the new ip on the node so it was just blank, which is when I decided to clean install the whole cluster.

[valkenburg-prevue-ch]

hi all,

since the problem is that some ips from hetzner are simply blocked, this is imho an issue that is not solved and is not going away. So I spent the day looking into this: https://github.com/rpardini/docker-registry-proxy
I think I have some success with it.

I have a cx11 instance up, on which I have a docker-compose setup (because it's just a few apps, kubernetes would be overkill), which hosts static files (haha, massive speedup for microos images), and which hosts that docker registry proxy.

For this to work, I had to make two modifications to kube-hetzner: added a variable "additional_k3s_environment" with which you can enable CONTAINERD_HTTPS_PROXY env vars, which are needed as per https://docs.k3s.io/advanced#configuring-an-http-proxy , and preinstall_exec which lets you inject arbitrary shell commands before k3s gets installed, in my case necessary for installing the ca.crt from the proxy.

It works. And it saves a lot of bandwidth!

Now, my worry is that these additions are a bit of an overcomplication. The minimum change in order to be able to do this, are the two changes I explained above (I can push an MR if you want), but if we want this to work out of the box and be user friendly, either one would have to run the proxy on one of the machines, or one would have to create an additional cx11 instance in the cluster, which is a pain when you are still in the 10-server limit of the first months of your hetzner account. On the other hand, that latter scenario does have the benefit that the proxy can be part of the private network, and you can easily shield it from the outside world using the firewall. Also, then we could consider airgapping things...

Thoughts?

[mysticaltech]

@valkenburg-prevue-ch Please MR, this is super awesome! I will activate the "sponsor" in the repo right away, and I am sure we can raise at least 3 euros a month for a CPX11 node, and if not, I will assume the cost myself while we wait for donors! 🚀 🙏

[mysticaltech]

@valkenburg-prevue-ch On second thought, the thing that worries me is that this will proxy all container-pulling requests to us. This is not ideal, especially for huge clusters, it's also a lot of responsibility.

Also, how is that different from the functionality already in place via the k3s_registries variable, which basically does the same thing, more or less?

[mysticaltech]

So hopefully, as shared below a fix is incoming! However, definitely having a general HTTPS proxy feature would prove very useful IMHO. Please let me know what you think 🙏

[valkenburg-prevue-ch]

Yeah, agreed that a centralized public repo is not a solution.

The k3s_registries variables lets you define actual mirrors one by one, and setup auth etc. I use it. But this docker-registry-proxy is literally a proxy, so you do not need to modify any image references anywhere, you only need to route all your container pulls through the proxy. And this you achieve cluster-wide with the HTTP(S)_PROXY environment variables. Caching is configured by whitelisting domains for caching, so by default things are not cached. I believe this would also at once solve the microos download issues.

I'll make a PR for the two variables, and maybe later have time to make an optional proxy node as part of the cluster, which could proxy everything.

[mysticaltech]

Ok, I understand better now, thanks for clarifying! Sounds great.

[mysticaltech]

Folks, thank goodness, they have identified the issue at the source and are working on a fix! kubernetes/registry.k8s.io#138

[valkenburg-prevue-ch]

Hm, it does sound promising, however, I have the same node also being blocked by ghcr.io, github's registry.

[mysticaltech]

Damn! Please do open a GitHub support ticket, this is not normal or wanted.

[mysticaltech]

@valkenburg-prevue-ch @aleksasiriski Actually folks, using the k3s_registries variable, we can just proxy everything to a free instance of jfrog. I haven't tried it yet, but will do ASAP. Please do the same on your end if you can.

[mysticaltech]

If it works, we just replace the links during the kustomization and boom, problem partially solved. At least the cluster will deploy well and be fully operational! And later those who want can use k3s_registries post-deploy with an array of options.

[zek]

I'm not sure if I did it correctly but ...

Looks like jfrog ips are banned too.

[mysticaltech]

@zek FYI, see the chosen answer for this thread, the best way now is just to open a support ticket with Hetzner, they usually reply fast. They take care of it, by probably rotating the IP of your node (that's my supposition at least). Please try and let me know if they solved it for you!

[zek]

@mysticaltech I was looking for a permanent solution and solved it with my self-hosted mirror. I'm recreating the cluster time to time so I don't wanna create a support ticket every time.

See https://gist.github.com/giggio/2ebdb2ad98c8ab7a003396f733ec6d61

[mysticaltech]

Thanks for sharing @zek!

[valkenburg-prevue-ch]

Far from ready for a PR, but here is a branch where I try to get a container proxy integrated into the project: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/feat/integrated-proxy

Please test it if you have time. Set this:

module "kube_hetzner" {
  ...
  enable_proxy_node = true
}

Beware:

• the startup time is doubled compared to without proxy, because first the proxy is built with microos, and only after that the rest of the cluster is build. After the proxy is built with microos, it downloads the microos image again (yeah, silly), and then serves it to the rest of the cluster. So the number of downloads is a constant 2, no matter whether you have a single or a million nodes.
• if your proxy gets a bad public ip, your cluster will be stuck.
• the configuration of the proxy is for the moment hard-coded, in the environment variables in docker-compose.yaml.
• I don't know why, but the init.sh script fails sometimes, and then it succeeds if you simply do another terraform apply.
• At this moment the proxy does not seem to be the silver bullet I was hoping for, some images still have issues, getting 404 codes (but this is only at registry.k8s.io, so I hope that is really a problem at the server).

For fun, ssh into the proxy machine, and to this:

cd /var/proxy-runtime/
docker compose logs -f

and shit back with some popcorn and watch the "upstream_cache_status":"HIT" lines passing by.

[mysticaltech]

Yeah, honestly, it's a very unfortunate situation. But we'll figure it out. As @ifeulner said, a container registry/proxy is a must anyways for production workloads, and if it does caching too, it would solve your making-the-problem-worse worries.

[valkenburg-prevue-ch]

Yes, I agree. I think, though, we should try to make is as obvious and easy as possible for users. So, since maintaining any solution inside the project is bad for the maintainability of the project, I hope we can write a thorough guide in the documentation and some big warnings in the kube.tf.example when people want to run without any proxy. It's like selling a gun and not warning it can kill.

[mysticaltech]

💯

[mysticaltech]

The solution. We just add another helm chart config, with a flag and done. #505 (reply in thread)

[mysticaltech]

I had a look at integrated_proxy, very interesting how you did it with docker-compose and a new proxy module. But it does add some complexity, as you warned us too. Let's keep this branch for reference, and it could be a plan B.

The latest proposed solution just gets installed as yet an additional helmchart config. I will give it a shot ASAP.

[valkenburg-prevue-ch]

Well, you see, the replacing of links is something I'm worried about. It's unstable and hence high maintenance. From the link you posted to the k3s docs about k3s_registries, I understand that mirroring is actually much easier, and you don't need to replace any links at all. It goes down the list looking for mirrored versions of the images. I would like to understand how that works, though, how you setup such a mirror and how you populate it. This mitm proxy is super easy as it populate everything on demand. Hopefully jfrog or harbor can do that?

…

-------- Oorspronkelijk bericht -------- Op 26 jan. 2023 21:24, schreef Karim Naufal :

If it works, we just replace the links during the kustomization and boom, problem partially solved. At least the cluster will deploy well! And later those who want can use k3s_registries with an array of options. — Reply to this email directly, [view it on GitHub](https://github.com/orgs/kube-hetzner/discussions/505#discussioncomment-4791186), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A3W7X2JTAACD7X33G5PUXTDWULMPPANCNFSM6AAAAAATYQFRJ4). You are receiving this because you were mentioned.Message ID: ***@***.***>

[mysticaltech]

It would be a safe bet to say that all k8s.registry.io images are on quay.io, but the location folder such as quay.io/k8scsi/ is not predictable (so we can't use the patterns in k3s_registries, for quay at least, hence for that particular instance I prefer the good old sed just before kustomization. Trust me on that one, if quay works, it will be smooth and clean! :)

[mysticaltech]

Ok, just tried pull quay and it requires auth. Too bad. We are left with dockerhub for that one, unless we can find another registry.

[mysticaltech]

Ok, turns out DELL has a platform called ObjectScale, and they seem to mirror registry.k8s.io on dockerhub, meaning we can use k3s_registries to proxy all requests from registry.k8s to dockerhub! 🚀

[mysticaltech]

Did some testing, and unfortunately, hitting docker.io directly won't work because most repos do not have up-to-date tags, and none have all the images.

Used this, but am not even sure of the format, also tested with crioctl -D pull <image-name> to triple-check, as it takes into account the rewrites (under the hood, no output is shown).

  k3s_registries = <<-EOT
mirrors:
  k8s.gcr.io:
    endpoint:
      - "https://registry-1.docker.io"
    rewrite:
      "autoscaling/(.*)": "bitnami/$1"
  registry.k8s.io:
    endpoint:
      - "https://registry-1.docker.io"
    rewrite:
      "sig-storage/(.*)": "objectscaling/$1"
  EOT

[mysticaltech]

So that avenue dried up for me.

[mysticaltech]

When a service is free, I do not want us to provide and maintain servers and infrastructure.

All is in your hands @valkenburg-prevue-ch. Providing an alternative solution to Jfrog proxying would definitely be great, like a module, offered in a new repo, that deploys a server that can be attached to a local network, and provide proxy services.

The only requirement would be that it lives in the same Hetzner project of course so that the network is visible. So, it would have the network name as a variable of course. And another variable would be the config itself of the tool, with defaults that work for problematic registries that we know of.

If when they deploy they get a bad IP, we have to check at the end and output the value of SUCCESS or FAILURE. And if Failure, we explain the procedure of destroying, reserving the IP (great tip from @aleksasiriski), and applying again.

Now if somehow this can work with k3s_registries, wonderful! 🥳

[aleksasiriski]

Ohhh I see it now.. That's weird? I tested autoscaling with

source          = "kube-hetzner/kube-hetzner/hcloud"

and it works now @mysticaltech

[valkenburg-prevue-ch]

That does make sense, as you use the published release 1.8.4, which has indeed the right link. But apparently @mysticaltech has built that off a branch that is not in this repository. I'm guessing.

[aleksasiriski]

I fixed it in master and staging, was a typo probably but idk how the correct link found itself in terraform source

[valkenburg-prevue-ch]

Back to the gitlab as a dependency proxy, here's their roadmap: https://about.gitlab.com/direction/package/#dependency-proxy

They currently only mirror docker.io, and they state themselves that jfrog artifactory and sonatype nexus are more mature, and do mirror other repo's too..

[mysticaltech]

I had forgotten to push the link fix to master, thanks again @aleksasiriski, and sorry for the confusion folks. About staging, I will merge this weekend, just want to test it first.

About the proxy, thanks for sharing @ifeulner, nexus seems great, and their OSS version seems to support proxying, so that is the sweet spot IMHO. As described above, deployed as a separate node (from a separate repo), that can be attached to the local network of the cluster.

As for forcing users, I think it's not needed, but we will strongly encourage them in the kube.tf.example (k3s_registries section) to use one of the SaaS proxying options or use our own repo to deploy it on an instance as small as cpx11.

[mysticaltech]

Just so you know, everyone, Hetzner, and the Kubernetes team are both aware of the issue and Hetzner seems to accept support requests for that issue (over at https://console.hetzner.cloud/support). They may have a way to clear the IP or most probably just assign a new one to the node!

See kubernetes/registry.k8s.io#138 (comment) and hetznercloud/csi-driver#373 (comment)

It does require opening a support ticket over at https://console.hetzner.cloud/support.

---

That said, I am still curious to try Trow, the in-cluster image registry (below is a fork with the functionality we need that is being merged into the main project here Trow-Registry/trow#329, however, we can just use the fork for now as it's well maintained by an active dev):

https://github.com/Extrality/trow/blob/main/docs/USER_GUIDE.md#proxying-other-registries-and-mutatingwebhook
https://github.com/Extrality/trow/blob/main/charts/trow/values.yaml#L30

[mysticaltech]

@schlichtanders There is a another possible solution, though not tried, is to use an egress gateway with cilium. We support that. Normally it should work. Please do give it a shot, see kube.tf.example, and we probably have an example in the readme section (just search the repo).

[mysticaltech]

Basically, you set the egress on a node that is not blacklisted, and boom it should work for all other nodes!

[mysticaltech]

FYI, people, latest simple and beautiful solution https://github.com/spegel-org/spegel

[WebSpider]

Oooh, that's a nice solution!

[mysticaltech]

Also can be enabled via k3s directly, see https://docs.k3s.io/installation/registry-mirror. We will also add explicit support for it.

[zek]

I followed following gist to create a mirror on one of my servers.

https://gist.github.com/giggio/2ebdb2ad98c8ab7a003396f733ec6d61

Finally it works!

[mysticaltech]

Folks, new solutions showed up, the best seems to be to just install Spegel or peerd. The former seems simpler.

[valkenburg-prevue-ch]

And spegel is in fact already embedded in k3s: https://docs.k3s.io/installation/registry-mirror?_highlight=spegel
🤯

[valkenburg-prevue-ch]

Which makes it entirely justified, imho, to enable it as part of the default kube.tf (either opt-in or opt-out).

[mysticaltech]

Wow! What a great finding, most definitely. Finally... We will be able to put this issue behind us. If you feel like it, PR most welcome mate 🫡

[schlichtanders]

This sounds unbelievable! Would this also solve blacklisted IPs?

[mysticaltech]

@schlichtanders It is THE fix for that, if you have more than one node that is (given that it's quite unlikely to have two blacklisted IPs in a row)! 🥳