Manually Validate Kubeadm 1.7 -> 1.8 Upgrade Path

[jessicaochen]

Automated upgrade testing is tracked by: #402

Meanwhile, I will be tracking manual upgrade testing here so we get some signal for 1.8 release.

[jessicaochen]

/assign jessicaochen

[jessicaochen]

Testing kubernetes commit a3ab97b7f395e1abd2d954cd9ada0386629d0411

Manual Upgrade: PASS (after working around [Issues 1-3])
Conformance After Upgrade: PASS

[Issues]
[1] Kubernetes Anywhere - kubernetes-retired/kubernetes-anywhere#456
/etc/kubeadm/kubeadm_init_params.txt was not created and populated

[2] Kubeadm - #470
sudo kubeadm config upload from-flags --token 7e6988.7243c1800f956615 --apiserver-bind-port
443 --apiserver-advertise-address 35.188.180.6
unable to fetch file. URL: "https://dl.k8s.io/release/stable-1.8.txt" Status: 404 Not Found

[3] Kubeadm - kubernetes/kubernetes#53043
sudo kubeadm upgrade apply latest-1.8 -y -f
[preflight] Running pre-flight checks
couldn't parse version "latest-1.8" as a semantic version

[Details]
Kubeadm: gs://kubernetes-release-dev/bazel/v1.8.0-rc.1.6+a3ab97b7f395e1/bin/linux/amd64
Kubelet: gs://kubernetes-release-dev/bazel/v1.7.7-beta.0.29+1a9e43f9d53b3b/bin/linux/amd64
Kubernetes Control Plane Inital Version: latest-1.7 (d3faa3f8f2e85c8089e80a661955626ae24abf80)
Kubernetes Control Plane Final Version: v1.8.0-rc.1 (c8a2429cb355d357d90174d3744a930dbd0bbff6)
(Kubernetes Anywhere Version: 646e7939c73b2a793e0f10b8fec12d8bc3b2292fq)

[jessicaochen]

@pipejakob is Issue 2 expected? I think you had some ideas on a similar one recently.
@luxas is issue 3 expected?

[pipejakob]

Oh interesting. On (2), it is expected that stable-1.8 won't resolve at this point (since we haven't released any stable versions for 1.8 yet), but it is unexpected to me that this command would actually need to resolve the version. This might just be some lazy code that always tries to resolve the --kubernetes-version flag (which defaults to stable-1.8) as soon as possible, even before we know we need it. A quick workaround would be to explicitly say --kubernetes-version=latest-1.8 instead. This shouldn't be a problem after we officially release 1.8.

[jessicaochen]

Huh, I am surprised that "stable" was not chosen instead. That would always resolve to the latest stable available. @luxas do you think we could use stable as the default in the future so we do not break tests when preparing for a major release? (I am referencing change kubernetes/kubernetes@81840b3)

[pipejakob]

@jessicaochen The idea is that, if you happen to try using an older version of kubeadm, it likely has no idea how to install newer versions of the control plane components (which have different flags). If you run an old 1.6.3 version of kubeadm, and it defaults to just getting the latest stable, which may be 1.8.3 in the future, the code it has to try to configure the control plane and create the necessary API objects is very likely not going to work because so much has changed since the kubeadm code was written. It is only expected to work for 1.6.x releases, so it defaults to stable-1.6 instead.

[jessicaochen]

This does not quite make sense either. If you have an old 1.6.3 version of kubeadm, is it expected to be able to install 1.6.5 if 1.6.5 becomes stable-1.6?

[pipejakob]

Yeah, if it's an easy fix, that sounds like a good approach to me. It definitely feels wrong to resolve the flag and not even use it (unless we actually use it for version checking during this phase, but I don't think we do).

[luxas]

@jessicaochen Both issue 2 & 3 are expected.
2: I'd really want to populate stable-1.8 (but not stable) when RCs are cut. That is not the case yet. Right now we have a small period of time (between RC cut and official release), where you have to set this to latest-1.8 or something else explicitely. See: kubernetes/kubernetes#52085

3: We deliberately chose to not be able to resolve CI labels for kubeadm upgrade apply. However, we've discussed it since the initial design phase, and will allow CI labels in v1.9: kubernetes/kubernetes#53043 for more consistent UX. Initially, we feared that the results might be unexpected, and being explicit with the version you want to upgrade to made sense. In the end, it might just cause confusion.

If you have an old 1.6.3 version of kubeadm, is it expected to be able to install 1.6.5 if 1.6.5 becomes stable-1.6?

Yes

BTW; we have the upgrade tests up and running now as well: https://k8s-testgrid.appspot.com/sig-cluster-lifecycle#kubeadm-gce-upgrade-1.7-1.8 🎉

@jessicaochen we should try to build the debs as well and update the kubelets in the cluster after the control plane is successfully upgraded with kubeadm upgrade apply

[pipejakob]

There's now an issue for (2): #470 (thanks, @medinatiger!).

[luxas]

This upgrade path has now been manually validated, and we have green e2es running.