Fix --kubelet-certificate-authority not defined

[woopstar]

--kubelet-certificate-authority is currently undefined on the kube apiserver. Enabling it causes the following error to happen, as nodes are not in the signed ip range in certs:

Error attaching, falling back to logs: error dialing backend: x509: certificate is valid for 10.50.63.11, 10.50.63.11, 10.50.63.12, 10.50.63.12, 10.50.63.13, 10.50.63.13, 10.248.0.1, 10.50.63.10, 127.0.0.1, 10.50.63.10, not 10.50.64.11
Error from server: Get https://10.50.64.11:10250/containerLogs/default/load-generator-5c4d59d5dd-mcg9h/load-generator: x509: certificate is valid for 10.50.63.11, 10.50.63.11, 10.50.63.12, 10.50.63.12, 10.50.63.13, 10.50.63.13, 10.248.0.1, 10.50.63.10, 127.0.0.1, 10.50.63.10, not 10.50.64.11

Looking at the openssl.conf file on a master, it reveals that no node ip adresses are actually in any certs for the nodes:

# cat /etc/kubernetes/openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = localhost
DNS.6 = odn1-kube-cluster01-master01
DNS.7 = odn1-kube-cluster01-master02
DNS.8 = odn1-kube-cluster01-master03
DNS.9 = odn1-kube-lb.privatedns.zone
IP.1 = 10.50.63.11
IP.2 = 10.50.63.11
IP.3 = 10.50.63.12
IP.4 = 10.50.63.12
IP.5 = 10.50.63.13
IP.6 = 10.50.63.13
IP.7 = 10.248.0.1
IP.8 = 10.50.63.10
IP.9 = 127.0.0.1
IP.10 = 10.50.63.10

Like here, 10.50.64.11 is missing which is a node, not a master.

[mattymo]

This won't scale. It means recreating kube-apiserver cert every time you want to add/remove a node. This is a destructive process which means forcing regeneration of service account secrets and restarting affected pods.

[woopstar]

Good point. Didn't think of that. Any suggestion

[woopstar]

Actually, we should only generate a unique certificate per node and apply a openssl.conf ony with the node's ip/dns in it. And then sign it by the CA file.

That would be the right way to do it

[NicolasT]

Just ran into a similar problem, trying to replace a 'crash-and-burn'ed master node by a 'new' one, but using a different IP:

fatal: [node-04]: FAILED! => {"changed": false, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/node-crb.yml) command (rc=1), out='', err='Unable to connect to the server: x509: certificate is valid for 10.100.2.70, 10.100.2.70, 10.100.1.44, 10.100.1.44, 10.100.2.160, 10.100.2.160, 10.233.0.1, 127.0.0.1, not 10.100.2.142\n'"}

Indeed, the list of 'valid' IPs are the original nodes, including the crashed one, and not the new one.

Creating a single CA once, then having per-server keys & certs, generated when necessary or even for every run, seems like a more correct approach, no?

[woopstar]

You are totally right. I just need to update the PR to do so. Or feel free to make a PR that does and we can close this.

[andrewb3000]

Have this been implemented in some other PR maybe?

[woopstar]

Have this been implemented in some other PR maybe?

--kubelet-certificate-authority is defined after we switched to kubeadm-deployment