clarify kubelet upgrade process

[liggitt]

Follow up from #11060, tracked in #12329

Upgrade process for kubelet is not sufficiently clear in user-facing documentation:

• add details to kubelet upgrade procedure (whether drain is required, whether skip-level kubelet upgrades are supported, etc) @kubernetes/sig-node-pr-reviews @kubernetes/sig-storage-pr-reviews

Page to Update:
https://kubernetes.io/docs/setup/version-skew-policy/

[roberthbailey]

Is this about in-place updates of the kubelet? Do the kubeadm upgrade tests cover that scenario? The kube-up GCE upgrade tests just replace machines running the old kubelet with machines running a newer one, bypassing the in-place upgrade questions here.

Unless or until we have testing for in-place upgrades, the conservative answer is that the upgrade process for a kubelet is to provision a new machine with the desired kubelet version.

[liggitt]

Is this about in-place updates of the kubelet?

Yes

Do the kubeadm upgrade tests cover that scenario?

I don't know. @kubernetes/sig-cluster-lifecycle, @kubernetes/sig-testing?

[neolit123]

Do the kubeadm upgrade tests cover that scenario?

yes, but our tests are failing due to problems with the upgrade framework in k/k (e.g. ginkgo skipping); possibly some other reasons too. the current tests are pretty much unmaintained and there are plans to replace them with something else next cycle, hopefully.

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes

we do recommend draining in our 12 -> 13 upgrade process.
skipping minor versions is claimed as unsupported. (edit: not claimed -> claimed)

[liggitt]

cordon -> drain -> upgrade kubelet -> uncordon is the informal guidance I've seen until now. There may be other deployment-specific reasons to destroy and rebuild nodes (node-level improvements that only take effect for new nodes, etc), but from the kubelet's perspective, drain has been sufficient, as far as I know.

[imkin]

/wg lts

[sftim]

/language en

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sftim]

/kind feature
/priority backlog

[bowei]

cc: @freehan

[sftim]

/remove-lifecycle stale

[liggitt]

There is evidence of users upgrading kubelets between minor versions without draining pods (kubernetes/kubernetes#84443). If that is required, the kubelet upgrade docs need to be made explicit.

[BenTheElder]

Is it required? Do we have a formal decision on that?

[dlipovetsky]

Please, let also us address whether drain is required when upgrading between patch versions, e.g., 1.17.0 to 1.17.1. These upgrades are, arguably, more frequent than upgrades between minor versions, and therefore users have a greater incentive to skip drain.

[neolit123]

WRT workload stability, the process is the same for PATCH iterations, so a drain would be required there too. one potential difference with MINOR updates is the CPU checkpoint format of the kubelet is not supposed to change on PATCH iterations.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[detiber]

/remove-lifecycle stale

[liggitt]

/assign @derekwaynecarr @dchen1107

Routing to sig-node leads. If we require draining nodes before minor version upgrade or reconfiguration (and that is required, as far as I can tell), that needs to be made explicit

[sjenning]

My opinion, a cordon -> drain -> upgrade -> uncordon path is the safest thing to document for all situations. We should be able to do patch level upgrades (z-stream, in x.y.z) without draining but, imho, there is no point in complicating the guidance.

[dchen1107]

cordon -> drain -> upgrade kubelet -> uncordon is the only path supported by SIG Node today. In the past, there were efforts to do in-place upgrade kubelet, including containerized kubelet by CoreOS team to simplify the upgrading flow, but none of them are officially supported by the community.

Let's make the upgrade flow explicit in the doc for now, but open for the enhancement.

[dlipovetsky]

Now that it's official, I'll work up a docs PR 🙂

[sftim]

/triage accepted

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sftim]

/remove-lifecycle stale

[liggitt]

cordon -> drain -> upgrade kubelet -> uncordon is the only path supported by SIG Node today

Opened #26098 to update the doc.

The cluster-upgrade doc already included this information

For each node in your cluster, drain
that node and then either replace it with a new node that uses the {{< skew latestVersion >}}
kubelet, or upgrade the kubelet on that node and bring the node back into service.