Unregister node from RKE2 after agent deletion

[zifeo]

Currently RKE2 keeps the agent as NotReady:

NAME                     STATUS     ROLES                       AGE   VERSION
rke-cluster-blue-001     Ready      <none>                      65m   v1.21.5+rke2r2
rke-cluster-green-001    NotReady   <none>                      65m   v1.21.5+rke2r2

[remche]

As stated in documentation, you need to manually drain and remove node before downscaling a pool nodes.
I did not find a clean way to do it, using ssh on server node was leading to chaotic behavior.
I will be really happy if a clean implementation is proposed though ;)

[zifeo]

@remche Yes, I've seen this. The issue is that even upgrades are not stable if there is no volume on agent nodes. Is there a reason/use case to use a non volume server or node?

[remche]

@zifeo I did not manage to reproduce this issue. Can you provide a sanitized configuration ?

Is there a reason/use case to use a non volume server or node?

In my use case I only use ephemeral volume for VM w/o any problem.

[zifeo]

@remche I have experimenting based on a different setup here, I will update if I find something stable and portable.

[remche]

@zifeo Nice, do not hesitate to contribute back ;)
Would be very happy to find a clean autoscaling method !

[zifeo]

As stated in documentation, you need to manually drain and remove node before downscaling a pool nodes.

This seems related to k3s-io/k3s#1264.

As for autoscaling, I would suggest something like orchestration_stack_v1 to keep a coherent Terraform state. This should work for simple autoscaling behaviours, but a custom/vanilla cloud provider could be written for advanced use cases.

[remche]

As for autoscaling, I would suggest something like orchestration_stack_v1 to keep a coherent Terraform state. This should work for simple autoscaling behaviours, but a custom/vanilla cloud provider could be written for advanced use cases.

I came to the same conclusions. Using Heat stack seems pretty hacky to me, I would prefer a custom cluster-autoscaler but it's more work :)

[zifeo]

@remche I will build a poc later to see how stable it could be. The issue with a custom autoscaler is the compatibility with the TF state. A remote & shared backend maybe but this seems even more hacky.

[remche]

My first though would be to use a remote state supporting locking. But I'm not sure there is a way to retrieve current backend configuration in data sources...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zifeo]

@zifeo Nice, do not hesitate to contribute back ;)
Would be very happy to find a clean autoscaling method !

Node deletion seems stable so far. I am happy to bring https://github.com/zifeo/terraform-openstack-rke2 over (merge all here), but the expose module interface is rather different. What is your point of view on this? This is why I chose to start from scratch originally.