Kubernetes 1.11 changes the way KUBELET_EXTRA_ARGS works

[geerlingguy]

Apparently in Kubernetes 1.11, the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file changed slightly; there is now a section:

# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet

And the contents of that file are, by default:

KUBELET_EXTRA_ARGS=

So the configuration in this role needs to be directed at /etc/default/kubelet instead of /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. 🤦‍♂️

[geerlingguy]

I ran into this because commands like kubectl logs [pod name] and kubectl exec were returning errors like:

root@master:/home/vagrant# kubectl logs a4d-nginx-67594d6bf6-9vfh6
Error from server (NotFound): the server could not find the requested resource ( pods/log a4d-nginx-67594d6bf6-9vfh6)

root@master:/home/vagrant# kubectl exec a4d-nginx-67594d6bf6-9vfh6 date
error: unable to upgrade connection: pod does not exist

[geerlingguy]

After a ton of fruitless searches, I found this comment: kubernetes/kubeadm#203 (comment) (which led me to reading the systemd unit file more closely). Testing a fix now... this will make the role incompatible with 1.10, at least if the user sets a value for kubernetes_kubelet_extra_args...

[geerlingguy]

After editing the correct file:

root@master:/home/vagrant# kubectl get nodes -o wide
NAME      STATUS    ROLES     AGE       VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
master    Ready     master    19m       v1.11.2   192.168.84.2   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-8-amd64    docker://18.6.1
node1     Ready     <none>    18m       v1.11.2   192.168.84.3   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-8-amd64    docker://18.6.1
node2     Ready     <none>    18m       v1.11.2   192.168.84.4   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-8-amd64    docker://18.6.1

[geerlingguy]

And now logs and all the other commands work correctly.