Is it safe to switch off master and worker servers during Hetzner network maintenance?

[Tom-Ravn]

Hi,

Tomorrow Hetzner is doing network maintenance in our region for about 1 hour. Last time they did this, it caused issues with our k3s cluster, cluster created a new node, moved all pods to it, and we had to reset IPs on our CDN. It was quite chaos.

I wonder how to survive this one hour better this time. Maybe we can just switch off worker servers and then master server and wait. I wonder what is recommended approach here. Worker pods are using connected disk volumes. The cluster is very small so far, just one master and one worker.

Thank you for advice.

[vitobotta]

Do your clusters have a single master?

[Tom-Ravn]

I know Vito, that is defnitely plan for future. But I think it does not answer my question regarding tommorow maintenance. I just want to avoid that little havoc it caused last time. If I switch it off, will everything come back ok after switching it on?
Or is there better way to handle such situations like hetzner's maintenances?

[vitobotta]

If you gracefully shut the master down I think it should be fine, but I have never tried this with a live cluster, so I cannot guarantee that when you turn it back on everything will be fine. Sorry if I can't give you a more definitive answer.

[Tom-Ravn]

Thank you Vito, so it imply to me, that my whole reasoning is wrong. Maybe I should ask what is proper way to go through hetzner maitenances. I guess having 3 masters is good step #1 :). But what else, just wait a see what it cause? They recommend to disconnet all connected volumes during maitenance. So I guess let it be running, is not best solution too. Maybe better would be to undeploy all pods which needs to work with volumes and re-deploy later again.
Thank you for your time.

[vitobotta]

We always run everything in HA config, not just control plane but also databases and everything else. Tomorrow for example a volume used by one instance in a database cluster will be affected, and up to this morning it was the primary instance in the (Postgres) cluster, so the maintenance would have caused downtime. So I just promoted another replica to primary to change this Unfortunately this kind of maintenance is always tricky because it's not always easy to work around them.

[Tom-Ravn]

Thank you Vito for shared knowledge. So I might undeploy affected pod and wait it through.

[vitobotta]

Best of luck! I hope Hetzner doesn't do this too often.

[Tom-Ravn]

Thank you, yes I hope as well.
Anyway thank you for building such amazing tool. It is really useful.

Have a great day Vito and take care!

[vitobotta]

You too!