Support with node rolling upgrades / Cluster-API

[saamalik]

Hi Team,

Setting up Rook with Ceph RBD storage on K8s was quite simple! Thanks for building such an awesome experience.

I had a question on how to use Rook with K8s environments which perform rolling upgrades of the nodes. In my testing, provisioning a new cluster e.g: 1 CP/3 workers with Rook works great. However, when I upgrade K8s (e.g: 1.18.17 -> 1.19.9), as new nodes join the cluster, existing nodes are evicted and eventually deleted.

The MON and OSD pods for the older nodes are not removed, and they're forever stuck in a pending state because the nodeSelector is still targetting the old node.

I don't think this is a typical use-case - but is this supported by rook? Are there any other specific configuration other than removeOSDsIfOutAndSafeToRemove?

[travisn]

@saamalik If you're installing in an environment where nodes are often recycled, you will want to create a cluster based on the cluster-on-pvc.yaml example. This will allow the underlying storage to not be dependent on the local node, and allow the mon and osd daemons to move to new nodes.

[saamalik]

Awesome, thanks!

[batistein]

We have the use case that we are using cluster-api on a baremetal environment. How could we achieve a similiar result?

From my point of view there are three possibilities:

1. create your own pvc. However, I do not know now how this goes on baremetal so easily ... Because the PV provisioner is missing and this is actually the purpose of rook/ceph? @travisn maybe you have a hint?
2. we write a controller which interacts with the cluster-api and waits until rook/ceph has rebalanced the OSDs, data must be copied/written.
3. we write a controller which interacts with the cluster-api. When recycling nodes, we unmount the data partition/disks and create the new OS only on the partition/disk hosting the OS. The disks are then mounted again. Also here we have to wait until the OSDs are balanced, but the writing of the data is dropped, which speeds up the process.

[travisn]

@batistein When working in a bare metal environment you're not going to have a PV provider that is capable of moving a PV to a new node, or at least I'm not aware of one. So I can imagine two realistic ways to approach this with the non-PV approach in Rook:

1. If the disks can be unmounted from the decommissioned host and mounted to the new host, Rook can detect them when the new host comes up, and consume the same device with no loss of data. This just requires that the node be added to the same K8s/Rook cluster, and not a different cluster (Rook has a unique cluster ID that is important for all the Ceph daemons). If the node name changes, you may want to fix the host name in the Ceph CRUSH map. While this maintenance is in progress, you can prevent unnecessary backfill of data with the noout flag for the OSDs.
2. If the first option is not possible, when a node is recycled you would need to consider the OSDs on that node lost, remove the OSDs, and simply start a new node where new OSDs would be provisioned to fully restore the storage capacity and backfill the data. Of course, this is a much more disruptive operation.

[batistein]

thank you very much for the detailed answer. Is there a way to make this also automated / API? I think it comes down to the fact that we have to write a microservice that coordinates with the cluster-api.

[travisn]

Rook doesn't automate this since it doesn't have the full context to know if a node is lost, or whether devices could move to another host. Automating this would depend on the environment. I'd suggest testing whether this works for your environment, then you would need to look at automating it.

[batistein]

ok that's clear, I don't think I formulated the question correctly. What is the correct interface to change or delete something like the crush map or the OSD? Is the CEPH api the right way or should this be done in rook? Thanks a lot in advance!
For the problem, you mentioned the machine-delete-phase-hooks are the solution. The cluster-api has this needed informations about the nodes. So we can place a controller between which is doing the necessary steps on both sides...

[travisn]

The expected way to interact with the cluster is through the CRDs. For the CRUSH map, there are ways to setup the hierarchy with node labels. Is there something else you need to do with the CRUSH map?