Cluster API doesn't directly support certificate renewal

[vishu2498]

What would you like to be added (User Story)?

Cluster API should be having a direct role in renewing the certificates of a cluster in the cluster's lifecycle. In its reconcilation process, it should be checking that if the certificates are expired, it should automatically renew them.

Detailed Description

The certificates that are generated by kubeadm are valid for only 1 year.

Anyone can check for the certificate expiry by this command inside the control-plane node of a cluster:

kubeadm certs check-expiration

To renew the certificates in a cluster, the command directly provided by kubernetes is:

kubeadm certs renew all

However, the problem here is that user currently won't know when the certificates are going to expire. Even though the user gets to know them and does the cert renewal by themselves, the process is still not complete.

The reason is that after the cert renewal happens, we are greeted with this message from the output:

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Since these are core components of kubernetes and static pods too, just restarting the pods won't make them use the new certificates. For the static pods, we have to remove the files of these pods from "/etc/kubernetes/manifests/" directory, wait for a few seconds, let the pods get deleted, and put the files back in. This process should be done manually as suggested by official Kubernetes document: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal

Also, the cluster will have a little bit of downtime till the core components of Kubernetes get back up.

Challenges observed when the solution can be implemented:

• We may lose cluster access for a while.
• The certs should not be deleted in any case. Since as per current testing, the "kubeadm" command for certificate renewal is failing for creating new certificates if any of the certificate is not present.
• When the reconcilation process is occurring, if the process fails, the code should back up the manifests in the next reconcile.
• While the code does this operation, there may be a chance that the CAPI pod gets deleted in the middle of moving files in and out. If it recovers in-between and get into running state, it should continue the process. This means that each file should be handled individually. However, if it goes in "CrashLoopBackOff" state, cluster will not be recoverable from that state.

A few people have already made some bash scripts or golang code to perform this operation. Here's one reference for bash script:

• https://github.com/yuyicai/update-kube-cert

cc @snehala27 @sadysnaat

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[neolit123]

Cluster API should be having a direct role in renewing the certificates of a cluster in the cluster's lifecycle. In its reconcilation process, it should be checking that if the certificates are expired, it should automatically renew them.

note that this contradicts the original CAPI immutable machine design - i.e. you are not supposed to change a machine to renew its certificates or change it's state once it's added. instead, new machines can join the cluster before the certificates expire and old machines can be removed.

that said, there is a new feature group that is about to discuss in-place upgrades in CAPI. and if CAPI uses kubeadm upgrade to do the in-place upgrades, it will benefit from the certificate rotation that kubeadm upgrade does by default.

#9559
#9489
#7415

[killianmuldoon]

/triage accepted

Let's keep this open for reference for the in-place upgrades folks to discuss, and keep open or close as they'd prefer.

[fabriziopandini]

Isn't this what we are describing in https://cluster-api.sigs.k8s.io/tasks/certs/auto-rotate-certificates-in-kcp?

[oblazek]

Isn't this what we are describing in https://cluster-api.sigs.k8s.io/tasks/certs/auto-rotate-certificates-in-kcp?

Yeah, seems to me it's exactly that.

[fabriziopandini]

/close
as per comment above

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close
as per comment above

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.