pods get recreated daily

[mkreuzmayr]

I don´t know if this has to do someting with kube-hetzner in particular but I noticed the following today and I cannot think of what happened. Somehow every day all of my pods get recreated with the exception of some in the pods kube-system namespace, they only get restarted.

I have got three nodes and they are all up since 25d. My hetzner control panel also does not show any activities concerning node creation/deletion. It only shows service updates and volume/attach detaches.

I already inspected logs and events in kubernetes or if I have any jobs or tasks that could lead to that.

[mrhein]

EDITED

in short: microos tumbleweed means daily updates + kured => autopatching

microos tumbleweed means daily updates => https://ftp.gwdg.de/pub/opensuse/tumbleweed/appliances/
you can see the changes here: https://ftp.gwdg.de/pub/opensuse/tumbleweed/iso/

[mkreuzmayr]

Ok, that makes sense. If i have multiple replicas of my pods, will the service be available while updating or is there a short downtime?

[otavio]

This also concerns me. We are moving our services to this and the downtime is a bad thing for us.

[mrhein]

hopefully it should not be problem otherwise you need to check why it is a problem for your services in best practise world it shouldn't be a problem because only one node after one is rebooting

dDatabases (Postgres) also run without problems

[maaft]

I have the same issue, but in particular in combination with Hashicorp Vault (one of the most popular secret stores): After being restarted, the vault gets automatically sealed. And for security purposes, the unseal process must be manual.

Meaning, our corporate secret store (vault) would required manual intervention every day to stay functional.

Any idea what I can do instead? Could we do manual, scheduled upgrades of MicroOS instead? If so, how?

edit vault has an auto-unseal option. But it requires either AWS, GCP or Azure AD which I want to avoid. Or another self-hosted vault cluster. Chicken-Egg problem..

p.s. @mrhein What postgres operator are you using? Are you satisfied? I'm currently not sure if I'll go with bitnami or zalando.

[mrhein]

from architecture perspective you can solve it with a management cluster running rockylinux or simlar with your vault harbor and so on and your application cluster in a more updated way the problem is than you have 2 setups to manage and a more vulnerable platform

maybe hetzner will plugin one of this YubiHSM 2 if we ask them (atleast for metal server), because vault says...

Vault integrates with the YubiHSM 2 via PKS#11 for customers who need hardware backed security. The integration supports auto unseal, cryptographic key generation and management, & external entropy.

so or so the kured config should be edited
https://kured.dev/docs/configuration/#setting-a-schedule
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/kustomize/kured.yaml#L19

that should help you out maybe update times only 3 days in row in middle of the week 9 to 11 am

p.s. @maaft there are only one solution on the market that is great and that ist zalando => postgres opertor which spawns patroni => spilo for you

[maaft]

What exactly does make zalando operator "great" in your opinion? E.g. in comparison with the one from bitnami

Key differences from the top of my mind:

• zalando integrates wal-g for easy backups/recovery (for bitnami one has to figure out backups/restores themselves)
• zalando uses patroni / spilo; bitnami uses repmgr ==> no idea which is "better"
• bitnami uses pgpool-II instead of pgbouncer, which I need anyway for R/W splitting so I'd need to install it anyway on top of zalando operator
• bitnami seems to release more frequently than zalando
• bitnami docs seem to be more polished

anything else?

[mysticaltech]

Hey folks, just FYI one of our initial contributors @mnencia is one of the lead architects of cloudnative-pg.io, also an amazing pg operator I heard! :)

[maaft]

thx for that amazing shoutout! CNPG is a beast! <3

[mysticaltech]

@mrhein Exactly. This is by design. When an update comes, microOS applies in a new snapshot, then signals it to kured, which in turn drains the node and reboots to apply the new snapshot. If you are running in HA, >= 3 control planes, and > 2 agents, you should have 0 downtime.

Now @mkreuzmayr @maaft Note that you can disable this behavior if you choose so, there are variables for that, also see the manual way in the readme.

[maaft]

As soon as I changed that label manually, the k3s upgrade triggered.

@mysticaltech Might this be a general bug where we need a new issue? (automatic k3s upgrades not working when os upgrades are active)

[mysticaltech]

@maaft It makes sense... Damn! PR welcome for this urgent fix, please.

[Lennix]

Any updates to this?
I'm actually also trying to limit the upgrade schedule to once a week (during the weekend preferably) and reading this discussion made me look at my cluster and also notice that all my nodes have the kured=rebooting label...

[aleksasiriski]

Any updates to this? I'm actually also trying to limit the upgrade schedule to once a week (during the weekend preferably) and reading this discussion made me look at my cluster and also notice that all my nodes have the kured=rebooting label...

There's an example in kube.tf

  # If you need more control over kured and the reboot behaviour, you can pass additional options to kured.
  # For example limiting reboots to certain timeframes. For all options see: https://kured.dev/docs/configuration/
  # The default options are: `--reboot-command=/usr/bin/systemctl reboot --pre-reboot-node-labels=kured=rebooting --post-reboot-node-labels=kured=done --period=5m`
  # Defaults can be overridden by using the same key.
  kured_options = {
    "reboot-days": "su"
    "start-time": "3am"
    "end-time": "8am"
    "time-zone": "Local"
  }

If you want it during the whole weekend (if you have a lot of servers) set reboot-days to sa,su and remove other options. I personally set it to monday in the middle of the night

[mysticaltech]

@Lennix This has been fixed for a long time. But one of our oldest versions had a bug with the labels. You can fix it with the following procedure, which is quite simple #562 (comment).

To limit once a week, @aleksasiriski provided sound advice above. Because updates do happen but are saved in a new btrfs snapshot, so they only kick in upon reboot!

[mkreuzmayr]

@mysticaltech I have got 3 control planes with control plane scheduling enabled, will it still be HA?

[mysticaltech]

@mkreuzmayr Most definitely!

[mysticaltech]

If your cluster is not resource-strained with the 3 nodes up and can handle one node going down for a few minutes, you are HA!