--certificate-renewal true doesn't renew kubelet.conf

[jeanluclariviere]

What keywords did you search in kubeadm issues before filing this one?

kubeadm kubelet certificate renewal
If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

Bug report

Versions

kubeadm version (use kubeadm version):
1.16.7 > 1.17.5
1.17.5 > 1.18.3

Environment:

• Kubernetes version (use kubectl version): v1.18.3
• Cloud provider or hardware configuration: bare-metal
• OS (e.g. from /etc/os-release): RHEL 7.8
• Kernel (e.g. uname -a): 3.10.0-1127.10.1.el7.x86_64

What happened?

After upgrading from 1.16 to 1.17, and 1.17 to 1.18 with the --certificate-renewal true switch, kubelet.conf does not reflect the updated certificates, but all other configurations in /etc/kubernetes are updated, including admin.conf, controller-manager.conf, and scheduler.conf.

What you expected to happen?

kubelet.conf should update its certificate to reflect the renewed certificates.

In my testing, the kubelet.conf cert was set to expire on 05/19/2021 - after the upgrade, 05/19/2021 was still there, it should have been renewed.

How to reproduce it (as minimally and precisely as possible)?

sudo kubeadm upgrade apply v1.18.3 --certificate-renewal true

Anything else we need to know?

According to the documentation, there is a bug in versions below 1.17
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

Warning:
On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

I assumed this was true for kubeadm upgrade as well, as there is another warning in the section regarding automatic certificate renewal:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal

Based on the first recommendation, I looked at the following cert /var/lib/kubelet/pki/kubelet-client-current.pem but found that this had not been renewed, and this still had the expiration of 05/19/2021.

However, if I check the expiration date of the certs, I can see that the new date is correct for the apiserver-kubelet-client certificate.

$ sudo kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 11, 2021 18:39 UTC   361d                                    no
apiserver                  Jun 11, 2021 17:42 UTC   361d            ca                      no
apiserver-etcd-client      Jun 11, 2021 17:42 UTC   361d            etcd-ca                 no
apiserver-kubelet-client   Jun 11, 2021 17:42 UTC   361d            ca                      no
controller-manager.conf    Jun 11, 2021 17:42 UTC   361d                                    no

Furthermore, if I output a new kubelet.conf, the new expiration date is used: sudo kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf

Other:

We originally ran in to this issue when a year went by and our certificates automatically renewed. The kubelet service was failing to come up as a result of the certificate renewal. My assumption was because we were on a version lower than 1.17 this failed (as described in the warnings), but after upgrading to 1.17.5 with the --certificate-renew true (and upgrading to 1.18.3), this issue persisted. I'm concerned that our kubelet will break on all our environments in another year from now

[neolit123]

hello, the kubelet.conf used to include embedded client certificates that kubeadm's certificate renewal did not manage.

after 1.17 kubeadm started writing kubelet.conf to point to the client cert files that are automatically managed / rotated by the kubelet.

see the note: "On nodes created with kubeadm init, prior to kubeadm version 1.17..." here

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

does this provide enough context to you?

/triage support

[jeanluclariviere]

Thanks for the prompt response - I understand that, according to that warning, we should modify our kubelet.conf to point to the following certificates instead:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

But after upgrading from 1.17 to 1.18, kubelet-client-current.pem does not appear to have been renewed. If I inspect the cert, it still contains the old expiration date - even though the renewal date when listing the certificate expiration dates had been updated apiserver-kubelet-client Jun 11, 2021 17:42 UTC 361d .

[neolit123]

But after upgrading from 1.17 to 1.18, kubelet-client-current.pem does not appear to have been renewed. If I inspect the cert, it still contains the old expiration date - even though the renewal date when listing the certificate expiration dates had been updated apiserver-kubelet-client Jun 11, 2021 17:42 UTC 361d .

the kubelet will decide when to rotate its certificates as long as client certificate rotation is enabled (kubeadm setups the kubelet with client cert rotation always enabled). kubeadm upgrade does not trigger the rotation.

[jeanluclariviere]

Understood - thanks for the clarification!