Ability to promote or demotes roles of custom cluster node without removing and rejoining the node

[axeal]

What kind of request is this (question/bug/enhancement/feature request):

Feature Request

Steps to reproduce (least amount of steps as possible):

User would like the ability to promote a worker node, i.e. add the controlplane or etcd roles to it, or vice-versa, to remove the controlplane or etcd roles from a node with multiple roles, without having to remove the node from the cluster and rejoin it with the desired roles.

The use-case is for example where there is failure of an etcd node, and to restore HA the user wishes to add a new etcd node. As they run on baremetal and cannot quickly spin up a new instance, in this scenario they add a third etcd instance by removing an existing worker node and rejoining with the etcd role, which reschedules numerous workloads and requires more intervention that a simple 'promotion' would.

Request is therefore to be able to change roles of nodes dynamically, i.e. re-run the agent container with different roles and have cluster provisioning reconcile to this.

Environment information

• Rancher v2.2.7

gz#5720

gz#14737

[georgebarbarosie]

I found this rather enigmatic phrase in the RKE docs here:

It’s possible to assign any combination of roles to any node. It’s also possible to change a node’s role using the upgrade process.

But I haven't been able to understand how to do this by looking at the upgrade chapter of the RKE docs, and I'm not sure I understand if it's applicable at all to rancher-initiated RKE clusters.

[dbeniciorj]

I found this rather enigmatic phrase in the RKE docs here:

It’s possible to assign any combination of roles to any node. It’s also possible to change a node’s role using the upgrade process.

But I haven't been able to understand how to do this by looking at the upgrade chapter of the RKE docs, and I'm not sure I understand if it's applicable at all to rancher-initiated RKE clusters.

Indeed. After loosing a etcd node because of a brief communication problem with it, I tried to reboot it after some time and it came up as a worker. There was no automatic rejoining (after all it was the same node....it would be nice if it was recognized somehow and automatically fixed) , and in this case i dind't had a worker to promote.

Anyway I just had to run manually rke up --update-only --config XXXX.yaml and got from this

... to this:

[phoenix-bjoern]

While this issue is still open is there at least a set of manual steps to change node roles? I found this one in the official K8S docu, but not sure if that will work with RKE as well: https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

[alwinmark]

Since Rancher Server 2.6, Rancher Server sometimes believes there is only one node and assigns it all roles at the next provisioning run. Therefore all my assigned roles like etcd and controlplane are just broken and pods like coredns get scheduled on the wrong node leading to DNS Problems and downtimes in my cluster as described here:

https://rancher-users.slack.com/archives/C3ASABBD1/p1648816069203179?thread_ts=1648469004.301599&cid=C3ASABBD1

[nicole-rappe]

Bringing this back into the loop as I am running into this same issue and want to "promote" at least one more of my nodes to a control-plane and etcd role and cannot find documentation on how to do this anywhere. (It all keeps looping back to this message chain in GitHub)

[nicole-rappe]

Revisiting this to understand if anyone from the community has such a way to demote / promote the roles of an existing cluster node without me having to remove the node and rejoin it.

@cyberstrawberry101 The following worked for me:

• From the overview, select your cluster
• On the left panel, go to nodes
• Klick on the three dots on the right for the node you want to modify
• Then edit the yaml
• Roughly at line 22 you can edit the role of the cluster
• Paste whatever role(s) you want it to have:

node-role.kubernetes.io/controlplane: 'true'
node-role.kubernetes.io/etcd: 'true'
node-role.kubernetes.io/worker: 'true'

For me, this persist across reboots, takes immediate effect and is reflected on the nodes themselves when issuing "kubectl get nodes"

Edit: Using Rancher 2.6.10 in this case

[harridu]

I would be interested in changing the node roles on the fly as well. I am extending my clusters. Got additional worker nodes, and new hardware to run Control Plane and etcd on dedicated KVMs. The cp/etcd load should be migrated from the old nodes to the new KVMs.

[harridu]

@Ek1m5, what about all the annotations and finalizers and other stuff in the yaml file? Can you confirm this is automatically adjusted to match the new node role?