Ubuntu 18.04 v1.14.4 AMI filesystem has kubelet cert/key

[dlipovetsky]

/kind bug

What steps did you take and what happened:
I deployed metrics-server on an a CAPA cluster. The metrics-server get metrics from kubelets because their server cert was not valid for their respective host addresses:

E0821 18:05:08.439746       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-86.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-0-86.us-west-2.compute.internal (ip-10-0-0-86.us-west-2.compute.internal): Get https://ip-10-0-0-86.us-west-2.compute.internal:10250/stats/summary/: x509: certificate is valid for ip-172-31-93-7, not ip-10-0-0-86.us-west-2.compute.internal, unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-233.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-0-233.us-west-2.compute.internal (ip-10-0-0-233.us-west-2.compute.internal): Get https://ip-10-0-0-233.us-west-2.compute.internal:10250/stats/summary/: x509: certificate is valid for ip-172-31-93-7, not ip-10-0-0-233.us-west-2.compute.internal, unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-103.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-0-103.us-west-2.compute.internal (ip-10-0-0-103.us-west-2.compute.internal): Get https://ip-10-0-0-103.us-west-2.compute.internal:10250/stats/summary/: x509: certificate is valid for ip-172-31-93-7, not ip-10-0-0-103.us-west-2.compute.internal, unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-15.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-0-15.us-west-2.compute.internal (ip-10-0-0-15.us-west-2.compute.internal): Get https://ip-10-0-0-15.us-west-2.compute.internal:10250/stats/summary/: x509: certificate is valid for ip-172-31-93-7, not ip-10-0-0-15.us-west-2.compute.internal]

After opening an ssh session to the host, I noticed that the kubelet cert was much older than the cluster itself:

root@ip-10-0-0-103:/var/lib/kubelet/pki# ls -al
total 20
drwxr-xr-x 2 root root 4096 Aug 20 21:56 .
drwxr-xr-x 9 root root 4096 Aug 20 21:56 ..
-rw------- 1 root root 1159 Aug 20 21:56 kubelet-client-2019-08-20-21-56-22.pem
lrwxrwxrwx 1 root root   59 Aug 20 21:56 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2019-08-20-21-56-22.pem
-rw-r--r-- 1 root root 2205 Jul 12 19:36 kubelet.crt
-rw------- 1 root root 1675 Jul 12 19:36 kubelet.key
root@ip-10-0-0-103:/var/lib/kubelet/pki# openssl x509 -in kubelet.crt -text | grep 172
        Issuer: CN = ip-172-31-93-7-ca@1562960163
        Subject: CN = ip-172-31-93-7@1562960163
                DNS:ip-172-31-93-7

My guess was that the cert is baked into the AMI. I confirmed this by manually creating an instance using the AMI:

root@ip-10-0-1-142:/var/lib/kubelet# curl --silent --show-error http://169.254.169.254/latest/meta-data/ami-id
ami-0b7f66ff29f86637f

root@ip-10-0-1-142:/var/lib/kubelet# ls -lR
.:
total 12
-rw------- 1 root root   62 Jul 12 19:36 cpu_manager_state
drwxr-xr-x 2 root root 4096 Jul 12 19:36 device-plugins
drwxr-xr-x 2 root root 4096 Jul 12 19:36 pki

./device-plugins:
total 0

./pki:
total 8
-rw-r--r-- 1 root root 2205 Jul 12 19:36 kubelet.crt
-rw------- 1 root root 1675 Jul 12 19:36 kubelet.key
root@ip-10-0-1-142:/var/lib/kubelet# 

What did you expect to happen:
No kubelet cert/key should be on the AMI filesystem.

Environment:

• Cluster-api-provider-aws version: v0.3.7
• Kubernetes version: (use kubectl version): v1.14.1
• OS (e.g. from /etc/os-release): Ubuntu 18.04

/cc @detiber

[dlipovetsky]

I can't exhaustively check all AMIs, but I can confirm this issue doesn't affect the Ubuntu 18.04 v1.15.1 AMI.

root@ip-10-0-1-215:~# ls -lR /var/lib/kubelet
ls: cannot access '/var/lib/kubelet': No such file or directory
root@ip-10-0-1-215:~# curl --silent --show-error http://169.254.169.254/latest/meta-data/ami-id
ami-0ed01a97c9f036c18

[detiber]

/assign
/lifecycle active

[detiber]

Closing as this was addressed with #1071 for newer k8s versions (v1.14.5, v1.14.6, v1.15.2, v1.15.3)
/close

[k8s-ci-robot]

@detiber: Closing this issue.

In response to this:

Closing as this was addressed with #1071 for newer k8s versions (v1.14.5, v1.14.6, v1.15.2, v1.15.3)
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.