Remove specific cp node from nodepool

[andi0b]

I have quite the disappointing situation that my first CP node has an IP address that is blacklisted by many cloud providers. So I would like to replace this node.

Is there any good possibility to do that? The node property (instead of count), which provides fine grained control only exists on agent nodes.

Can I delete the server in the Hetzner Console and will Terraform create a new one on next apply? @mysticaltech would that work, or will it just create chaos?

Or do I really need to create separate node pools with count=1 for all control plane nodes, to get some control over it?

I'm using metallb without a load balancer, which makes replacing all the nodes quite cumbersome, because I need to update a lot of DNS zones. And I'm not running dedicated CP nodes, I don't really see the point of running 3 CP nodes additionally to 3 worker nodes.

[mysticaltech]

@andi0b Two things, if you are suffering from the containers getting a 403 on pulling, there is a fresh and efficient solution for this. See the pinned discussion on the matter. That's a problem of the past now while this mitigation is installed, as it will pull directly through other working nodes.

Now if you prefer to still replace the ip, just make sure you are in HA, so at least 3 CP nodes, drain it, and cordon it, then with the hcloud cli or ui, delete it. Then create a static ip to capture temporarily the sick IP and then terraform apply to recreate the node with the new IP. That works well from my previous testing and is the whole purpose of having an HA setup. But as said above, if it's to solve container pulls, it's no longer needed.

[andi0b]

Thanks a lot for those tips!

In my case it's not the container pull, although there were some slow pulls recently. In my case the main OIDC endpoint that handles authentication is hosted in GCP (now I know it). Connectivity issues to the auth endpoint break a lot of workloads I'm running.

[mysticaltech]

Sorry for the late reply, don't hesitate to tag me directly next time (I get then get a notification). I would suggest you try with cilium egress gateway that we support. If the egress is from a whitelisted node, it should solve your issue.

[andi0b]

@mysticaltech np, my question was only about reinstalling the nodes, which is very specific to kube-hetzer. I don't want to bother you with general k8s questions that also ChatGPT can answer.

For now I just used annotations/anti-affinity to move the pods that need egress to the good nodes. This doesn't change a lot in my case compared to the added complexity of an egress gateway. All the pods that need egress don't need a lot of resources, so it's even better to just schedule them directly on the egress enabled nodes instead of creating additional complexity/network overhead.