kubeadm init specifying a version may result in a kubelet version mismatch

[innergy]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
v1.8.1

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration**:
  VM
• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.2 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.2 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a):
  Linux scspa0328555001 4.4.0-97-generic kubeadm init hangs on ppc64le  #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  kubeadm
• Others:

What happened?

We use kubeadm init's --kubernetes-version option to install specific versions of Kubernetes for testing purposes. Prior to running kubeadm, you have to install it. The kubeadm instructions ask you to install not just kubeadm but also kubectl and kubelet from the appropriate repo:

https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

The upshot of that is that you can end up with kubeadm, kubectl and kubelet versions that are (potentially much) newer than the kube containers that kubeadm runs based on the --kubernetes-version option.

This hasn't caused any noticeable issues in the past, but with the advent of 1.8 we started to notice that iSCSI mounts were suddenly failing. However, if we downgrade kubelet to match the version of Kubernetes we installed, it works fine.

What you expected to happen?

kubeadm should install the matching version of kubelet (and perhaps kubectl as well) during init and upgrade rather than asking the user to install it separately to avoid issues like this entirely. At the very least there should be a pre-flight check that warns the user that they chose a version of Kubernetes that's different from the version of kubelet on their machine.

How to reproduce it (as minimally and precisely as possible)?

If you follow the kubeadm instructions to the letter today but specify 'stable-1.7' as your version when you run kubeadm init, you will end up with:

kube-apiserver: 1.7.8
kube-controller-manager: 1.7.8
kube-scheduler: 1.7.8
kube-proxy: 1.7.8
kubeadm: 1.8.1
kubectl: 1.8.1
kubelet: 1.8.1

If you create an iSCSI PV after that and try to consume it from a pod, the kubelet will fail to mount it. If you then downgrade kubelet to 1.7.8, the mount succeeds.

Anything else we need to know?

[luxas]

@kad can you please add some preflight checks for init/join enforcing the kubelet policy?
(kubelet version can be in the 1.(X-2) - 1.X range where X is the minor version of the API server you're deploying)

[kad]

@luxas do we want to have it as warning ? in some cases e.g. 1.8 kubelet will work ok-ish with 1.7.x control plane.
Regarding join: I might be mistaken, but we don't have information about control plane version during join preflight checks.