Unable to upgrade to v1.10 using kubeadm

[rajansandeep]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
When trying to upgrade kubernetes from v1.9.6 to v1.10.0 using kubeadm, it throws a version error and doesn't upgrade.

# kubeadm upgrade apply v1.10.0
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to change the cluster version to "v1.10.0"
[upgrade/versions] Cluster version: v1.9.6
[upgrade/versions] kubeadm version: v1.9.6
[upgrade/version] FATAL: The --version argument is invalid due to these fatal errors:

	- Specified version to upgrade to "v1.10.0" is at least one minor release higher than the kubeadm minor release (10 > 9). Such an upgrade is not supported

Please fix the misalignments highlighted above and try upgrading again

What you expected to happen:

Successfully upgrade to Kubernetes v1.10.0 via kubeadm

How to reproduce it (as minimally and precisely as possible):

kubeadm upgrade apply v1.10.0

Anything else we need to know?:
This issue may be related to #61764. I raised a separate issue because this seems to be on a broader/general scope.

Environment: baremetal

• Kubernetes version (use kubectl version):

# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[rajansandeep]

/area kubeadm
/sig cluster-lifecycle

[lareeth]

Are you using RPMs? They haven't been updated yet, you need to update kubeadm to the same version you want to upgrade to.

Related: #61763 kubernetes/release#539

If not, download the latest binaries, and replace kubeadm with the latest version. Once upgraded, then replace the rest of the binaries, such as kubectl and kubelet.

[calebamiles]

/assign

[calebamiles]

rpms and debs have been published now, @rajansandeep

cc: @jdumars, @kubernetes/kubernetes-release-managers

[rajansandeep]

@calebamiles thank you!
But now, I see that the kubeadm upgrade plan and kubeadm upgrade apply hangs/times-out and is unsuccessful.

# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.10.0
[upgrade/versions] kubeadm version: v1.10.0
[upgrade/versions] Latest stable version: v1.10.0

[calebamiles]

Acknowledged, @rajansandeep. I think you the issue you identified is a question better handled by someone in @kubernetes/sig-cluster-lifecycle-bugs. Your issue might be tracked under kubernetes/kubeadm#740; and if so closing this issue will help keep the issue queue a bit more sane. Thanks!

[rajansandeep]

Yes, I think the issue you mentioned above covers this issue. Hence closing.