New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm init specifying a version may result in a kubelet version mismatch #53832

Closed
innergy opened this Issue Oct 12, 2017 · 5 comments

Comments

Projects
None yet
5 participants
@innergy

innergy commented Oct 12, 2017

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
We use kubeadm init's --kubernetes-version option to install specific versions of Kubernetes for testing purposes. Prior to running kubeadm, you have to install it. The kubeadm instructions ask you to install not just kubeadm but also kubectl and kubelet from the appropriate repo:

https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

The upshot of that is that you can end up with kubeadm, kubectl and kubelet versions that are (potentially much) newer than the kube containers that kubeadm runs based on the --kubernetes-version option.

This hasn't caused any noticeable issues in the past, but with the advent of 1.8 we started to notice that iSCSI mounts were suddenly failing. However, if we downgrade kubelet to match the version of Kubernetes we installed, it works fine.

What you expected to happen:
kubeadm should install the matching version of kubelet (and perhaps kubectl as well) during init and upgrade rather than asking the user to install it separately to avoid issues like this entirely. At the very least there should be a pre-flight check that warns the user that they chose a version of Kubernetes that's different from the version of kubelet on their machine.

How to reproduce it (as minimally and precisely as possible):
If you follow the kubeadm instructions to the letter today but specify 'stable-1.7' as your version when you run kubeadm init, you will end up with:

kube-apiserver: 1.7.8
kube-controller-manager: 1.7.8
kube-scheduler: 1.7.8
kube-proxy: 1.7.8
kubeadm: 1.8.1
kubectl: 1.8.1
kubelet: 1.8.1

If you create an iSCSI PV after that and try to consume it from a pod, the kubelet will fail to mount it. If you then downgrade kubelet to 1.7.8, the mount succeeds.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration**:
    VM
  • OS (e.g. from /etc/os-release):
    NAME="Ubuntu"
    VERSION="16.04.2 LTS (Xenial Xerus)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 16.04.2 LTS"
    VERSION_ID="16.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    VERSION_CODENAME=xenial
    UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a):
    Linux scspa0328555001 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    kubeadm
  • Others:
@innergy

This comment has been minimized.

Show comment
Hide comment
@innergy

innergy Oct 12, 2017

/sig cluster-lifecycle

innergy commented Oct 12, 2017

/sig cluster-lifecycle

@innergy

This comment has been minimized.

Show comment
Hide comment
@innergy

innergy Oct 13, 2017

/area kubeadm

innergy commented Oct 13, 2017

/area kubeadm

@dixudx

This comment has been minimized.

Show comment
Hide comment
@dixudx

dixudx Oct 13, 2017

Member

@innergy Kubelets (1.8.1) newer than the apiserver(1.7.8) are not well supported currently.

@luxas @liggitt To avoid this, shall we let kubeadm take control of kubelet to keep them in the same version. If so, I think kubelet can run in containerized mode, which will greatly simplify the version control. WDYT?

Member

dixudx commented Oct 13, 2017

@innergy Kubelets (1.8.1) newer than the apiserver(1.7.8) are not well supported currently.

@luxas @liggitt To avoid this, shall we let kubeadm take control of kubelet to keep them in the same version. If so, I think kubelet can run in containerized mode, which will greatly simplify the version control. WDYT?

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Oct 13, 2017

Member

Please move this issue to the kubeadm issue tracker.
Thanks!

Member

luxas commented Oct 13, 2017

Please move this issue to the kubeadm issue tracker.
Thanks!

@luxas luxas closed this Oct 13, 2017

@innergy

This comment has been minimized.

Show comment
Hide comment

innergy commented Oct 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment