kubelet certificate auto renew

[mrcule]

Hi all, I found that kubelet.crt will be expire after 1 year. Is the kubelet.crt automatically rotate? Do I need to add --rotate-certificates in /var/snap/microk8s/current/args/kubelet [1]?
What happen if it expired?

Also, is it possible to check when certificates expire [2]?

Thank you.

[1] : https://kubernetes.io/docs/tasks/tls/certificate-rotation/#enabling-client-certificate-rotation
[2] : https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

[balchua]

Ho @mrcule
There is a way to update the certificates.
Check the documetation here. https://microk8s.io/docs/command-reference#heading--microk8s-refresh-certs

If you are running a multi node cluster, you will have to remove and join back in the node.
Read the Note section.

[ktsakalozos]

@mrcule, some additional info on the kubelet certificate. Auto renewal (certificate rotation for the Kubelet) is not enabled by default in MicroK8s. To turn this on you will need to follow the the instructions in [1].

To check the expiration date of the certificate /var/snap/microk8s/current/certs/kubelet.crt you can openssl x509 -in /var/snap/microk8s/current/certs/kubelet.crt -text -noout. This certificate gets regenerated if you remove it and restart MicroK8s.

[1] https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/

[mrcule]

Hi @balchua @ktsakalozos.
Thank you for your response. I have an cluster running in an no internet access environment, I might not update the certificates in time. What happen if the certificates expired? Will the running pods get interrupted?

[mrcule]

@ktsakalozos, just tried to remove /var/snap/microk8s/current/certs/kubelet.crt and then restart Microk8s, but the kubelet.crt doesn't get generated. Tried microk8s refresh-certs as well but in vain.

Then I tried to run openssl req -new -x509 -nodes -sha256 -days 365 -key kubelet.key -out kubelet.crt -config csr.conf.template. It generates a new kubelet.crt, but I don't know whether it is correct to do this.

[mcfedr]

I think I'm seeing this issue, the certificate on port 10250 presented by kubelet, from /var/snap/microk8s/current/certs/kubelet.crt has expired.

the microk8s refresh-certs command doesnt have an option to refresh this cert.

I see the error in metrics server because of this

E1017 10:42:06.623537       1 scraper.go:140] "Failed to scrape node" err="Get \"https://10.2.0.14:10250/metrics/resource\": x509: certificate has expired or is not yet valid: current time 2022-10-17T10:42:06Z is after 2020-06-22T18:25:16Z" node="xxx"

The file doesn't appear to be created when deleted.

Error: failed to construct kubelet dependencies: error reading /var/snap/microk8s/4055/certs/kubelet.crt, certificate and key must be supplied as a pair

Until I deleted both the cert and the key and then restart microk8s and problem solved

[PRNDA]

Hi @balchua @ktsakalozos. Thank you for your response. I have an cluster running in an no internet access environment, I might not update the certificates in time. What happen if the certificates expired? Will the running pods get interrupted?

+1

I have a running multi-node cluster with a expired kubelet.crt too, I what to know if I recreate this cert manually, will the running pods get interrupted?

Additional questions:

I have found that /var/snap/microk8s/current/certs/front-proxy-client.crt and /var/snap/microk8s/current/certs/server.crt got expired too, what's these cert used for?

And, iff I refresh them by running microk8s refresh-certs and restart by running microk8s stop && microk8s start，will the running pods get interrupted?

[neoaggelos]

Hi @PRNDA,

Which MicroK8s version are you using? If you are using MicroK8s 1.22 or newer no workloads will be affected by rotating certificates. The certificates you mentioned are used for:

• kubelet.crt: Automatically created and managed by kubelet, used for the communication of kube-apiserver -> kubelet. I believe a side-effect of it being expired might be issues with kubectl logs commands.
• server.crt: Used by the API server and the cluster-agent. If expired, you should have general issues with components being unable to communicate with the API server due to TLS failures.
• front-proxy-client.crt: Used by the API server for the front-proxy aggregation layer. If any of these words do not ring a bell, you are not affected by it.

You can refresh server.crt and front-proxy-client.crt without any issues for your workloads, using the microk8s refresh-certs command.

[PRNDA]

Hi @PRNDA,

Which MicroK8s version are you using? If you are using MicroK8s 1.22 or newer no workloads will be affected by rotating certificates. The certificates you mentioned are used for:
...
You can refresh server.crt and front-proxy-client.crt without any issues for your workloads, using the microk8s refresh-certs command.

I'm using microk8s v1.23.16, thank you very much for your detailed explanation.
And one last question: If I use the microk8s refresh-certs command to refresh server.crt and front-proxy-client.crt, do I need to leave and rejoin the nodes to make the refresh take effect?

[neoaggelos]

do I need to leave and rejoin the nodes to make the refresh take effect?

Nope, no such need. The refresh certs command will generate the new certificates and restart the affected control plane services, without affecting workloads.

[PRNDA]

do I need to leave and rejoin the nodes to make the refresh take effect?

Nope, no such need. The refresh certs command will generate the new certificates and restart the affected control plane services, without affecting workloads.

I just read the microk8s refresh-certs command documentation, it does said that I need to leave and rejoin the nodes.

refs: microk8s refresh-certs command doc

I quoted it here:

Notes:
The CA should not be updated in a cluster with running workloads. Auxiliary certificates and credentials make use of the CA, so updating the CA in a live cluster will have unpredictable effects.

In a multi-node setup, nodes will need to leave and rejoin the cluster in order for new certificates to properly propagate.

[neoaggelos]

Hmm, interesting. OK, that note in the documentation does not look correct to me. I will make sure to double check and update it accordingly.

[PRNDA]

OK.
Thank you so much! Microk8s is fantastic.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.