Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade plan from 1.9.6 to 1.10.0 fails with "unable to decode config" #744

Closed
tangleloop opened this issue Apr 4, 2018 · 9 comments · Fixed by kubernetes/kubernetes#62568
Closed

upgrade plan from 1.9.6 to 1.10.0 fails with "unable to decode config" #744

tangleloop opened this issue Apr 4, 2018 · 9 comments · Fixed by kubernetes/kubernetes#62568
Assignees
@liztio
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone
v1.10

Comments

@tangleloop
Copy link

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

/tmp/kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}
    Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

  • Cloud provider or hardware configuration:
    a cluster of arm systems:
    https://ameridroid.com/products/odroid-hc2

  • OS (e.g. from /etc/os-release):
    Armbian Ubuntu
    NAME="Ubuntu"
    VERSION="16.04.4 LTS (Xenial Xerus)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 16.04.4 LTS"
    VERSION_ID="16.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    VERSION_CODENAME=xenial
    UBUNTU_CODENAME=xenial

  • Kernel (e.g. uname -a):
    Linux odroid01 4.9.61-odroidxu4 kubeadm init starts paused containers on ubuntu 16.04 #2 SMP PREEMPT Wed Nov 22 16:34:23 CET 2017 armv7l armv7l armv7l GNU/Linux

What happened?

kubeadm version 1.9.6 output looks appropriate, but it wants me to install version 1.10.0. When I run kubeadm upgrade plan with the new version, it generates an error. See output of both commands below.

kubeadm upgrade plan (v1.9.6)
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.9.6
[upgrade/versions] kubeadm version: v1.9.6
[upgrade/versions] Latest stable version: v1.10.0
[upgrade/versions] Latest version in the v1.9 series: v1.9.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 3 x v1.9.6 v1.10.0

Upgrade to the latest stable version:

COMPONENT CURRENT AVAILABLE
API Server v1.9.6 v1.10.0
Controller Manager v1.9.6 v1.10.0
Scheduler v1.9.6 v1.10.0
Kube Proxy v1.9.6 v1.10.0
Kube DNS 1.14.7 1.14.7
Etcd 3.1.11 3.1.11

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.10.0

Note: Before you can perform this upgrade, you have to update kubeadm to v1.10.0.

/tmp/kubeadm upgrade plan (v1.10.0)
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: could not decode configuration: unable to decode config from bytes: v1alpha1.MasterConfiguration: KubeProxy: v1alpha1.KubeProxy: Config: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in #10 byte of ...|reGates":"","healthz|..., bigger context ...|24h0m0s"},"enableProfiling":false,"featureGates":"","healthzBindAddress":"0.0.0.0:10256","hostnameOv|...

What you expected to happen?

that kubeadm upgrade plan would not generate an error

How to reproduce it (as minimally and precisely as possible)?

  1. Create cluster on v1.9.6 with kubeadm init & kubeadm join
  2. Download kubeadm 1.10.0 to /tmp/kubeadm and run /tmp/kubeadm upgrade plan
@timothysc
Copy link
Member

Thank you for reporting, this is know and we are fixing ASAP.
This is also related to #740

/cc @liztio

@timothysc timothysc added the kind/bug Categorizes issue or PR as related to a bug. label Apr 4, 2018
@timothysc timothysc added this to the v1.10 milestone Apr 4, 2018
@timothysc
Copy link
Member

This fix will need to be cherry picked.

@timothysc timothysc added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed triaged labels Apr 5, 2018
@liztio
Copy link

liztio commented Apr 5, 2018

I've tracked this down to this change: a field in kubeProxy changed from a string to a map.

@timothysc
Copy link
Member

@xiangpengzhao - The change to the config for unused fields cause an issue. We are planning a patch, but we need to generally fix the versioning issues, either through direct transform or eating all of the api-machinery.

@stealthybox
Copy link
Member

Similar to kubernetes/kubernetes#61764

@xiangpengzhao
Copy link

This issue is a dup of kubernetes/kubernetes#61764. Sorry for not noticing the potential risk of causing the issue when I sent the patch kubernetes/kubernetes#57962. I haven't found a proper approach to fix it yet. There is a workaround kubernetes/kubernetes#61764 (comment) for upgrading. I sent a PR kubernetes/kubernetes#61882 to update the CHANGELOG of 1.10.0. But yeah we should fix it a better way. I will think about @timothysc comment. Any more detailed suggestion would be appreciated :)

@timothysc timothysc mentioned this issue Apr 6, 2018
Closed
@timothysc
Copy link
Member

@xiangpengzhao we're working on a fix right now. We have a plan see slack.

@timothysc timothysc added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Apr 6, 2018
@liztio
Copy link

liztio commented Apr 6, 2018

/label active

@xiangpengzhao
Copy link

@timothysc cool! I didn't notice the plan :)

This was referenced Apr 10, 2018
Closed
Closed
@timothysc timothysc removed their assignment Apr 13, 2018
@liztio liztio mentioned this issue Apr 14, 2018
Merged
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Apr 18, 2018
Merge pull request #62568 from liztio/kubeadm-upgrade-same-version
edee03e 
Automatic merge from submit-queue (batch tested with PRs 62568, 62220, 62743, 62751, 62753). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Kubeadm upgrade same version

What this PR does / why we need it:

When kubeadm 1.10 came out, it inadvertently introduced a backwards incompatible config change. Because the kubeadm MasterConfiguration is written by the old version of kubeadm and read by the new one, this incompatibility causes the upgrade to fail.

To mitigate this, I've written a simple transform that operates on a map-based version of the config. This map is mutated to make it compatible with the new structure, then serialised to JSON and deserialised by the usual APIMachinery.

Because of complications with the multiple versions, this PR enforces kubeadm only being used to upgrade to kubernetes of the same minor and major versions.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes [kubeadm#744](kubernetes/kubeadm#744 (comment))
This PR is an alternate take on #62353. Instead of trying to gate migration on versions, this constrains kubeadm to only upgrade versions from the same major and minor versions.

Special notes for your reviewer:
```release-note
fixes configuration error when upgrading kubeadm from 1.9 to 1.10+
enforces  kubeadm  upgrading kubernetes from the same major and minor versions as the kubeadm binary.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liztio liztio

Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
v1.10
Development

Successfully merging a pull request may close this issue.

Kubeadm upgrade same version liztio/kubernetes
Kubeadm upgrade migrate liztio/kubernetes
5 participants
@timothysc @stealthybox @xiangpengzhao @liztio @tangleloop