kubeadm: apply "master" label/taint migration for 1.24 #107533

neolit123 · 2022-01-13T15:46:36Z

What type of PR is this?

/kind cleanup feature

What this PR does / why we need it:

adapt the kubeadm managed labels and taints to changes planned in release 1.24, as part of the effort to remove usages of the word "master".

short summary:

for new clusters, CP nodes will only have the new "control-plane" label ("master" label is no longer applied).
for new clusters, CP nodes will be tainted with both the "master" and "control-plane" taint
for upgraded clusters, "kubeadm upgrade apply" will remove the "master" label from CP nodes.
for upgraded clusters, "kubeadm upgrade apply" will apply "control-plane" taint (next to the existing "master" taint). the "master" taint will be removed in 1.25.

details from the KEP:
https://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint

1.24 = stage 2

## Design Details

The process will be broken into multiple stages:
- First - 1.20
- Second - Minimum deprecation period for GA features is 1 year.
Estimated 1.24, but may depend on user feedback.
- Third - one release after Second
- Fourth - one release after Third

### Renaming the "node-role.kubernetes.io/master" Node label

First stage:
- Introduce the "node-role.kubernetes.io/control-plane" label in parallel to
the "master" label.
- Announce to users that they should adapt to use the new label.

Second stage:
- Remove the "master" label and announce it to the users.

### Renaming the "node-role.kubernetes.io/master" Node taint

First stage:
- Introduce the "node-role.kubernetes.io/control-plane:NoSchedule" toleration
in the CoreDNS Deployment of kubeadm.
- Announce to users that they should do that same for their workloads.

Second stage:
- Add the "node-role.kubernetes.io/control-plane:NoSchedule" taint to Nodes.

Third stage:
- Remove the "node-role.kubernetes.io/master:NoSchedule" taint from Nodes.

Fourth stage:
- Remove the "node-role.kubernetes.io/master:NoSchedule" toleration in the CoreDNS
Deployment of kubeadm
- Announce to users that they should remove tolerations for the "master" taint in
their workloads.

Which issue(s) this PR fixes:

xref kubernetes/kubeadm#2200

Special notes for your reviewer:

Does this PR introduce a user-facing change?

ACTION REQUIRED: kubeadm: apply "second stage" of the plan to migrate kubeadm away from the usage of the word "master" in labels and taints. For new clusters, the label "node-role.kubernetes.io/master" will no longer be added to control plane nodes, only the label "node-role.kubernetes.io/control-plane" will be added. For clusters that are being upgraded to 1.24 with "kubeadm upgrade apply", the command will remove the label "node-role.kubernetes.io/master" from existing control plane nodes. For new clusters, both the old taint "node-role.kubernetes.io/master:NoSchedule" and new taint "node-role.kubernetes.io/control-plane:NoSchedule" will be added to control plane nodes. In release 1.20 ("first stage"), a release note instructed to preemptively tolerate the new taint. For clusters that are being upgraded to 1.24 with "kubeadm upgrade apply", the command will add the new taint "node-role.kubernetes.io/control-plane:NoSchedule" to existing control plane nodes. Please adapt your infrastructure to these changes. In 1.25 the old taint "node-role.kubernetes.io/master:NoSchedule" will be removed.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

https://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint

- Rename the function in postupgrade.go to better reflect what is being done. - During "upgrade apply" find all nodes with the old label and remove it by calling PatchNode. - Update health check for CP nodes to not track "master" labeled nodes. At this point all CP nodes should have "control-plane" and we can use that selector only.

- Update the markcontrolplane phase used by init and join to only label the nodes with the new control plane label. - Cleanup TODOs about the old label. - Remove outdated comment about selfhosting in staticpod/utils.go. Selfhosting has not been supported in kubeadm for a while and the comment also mentions the "master" label. - Update unit tests.

- Apply "control-plane" taint during init/join by adding the taint in SetNodeRegistrationDynamicDefaults(). The old taint "master" is still applied. - Clarify API docs (v1beta2 and v1beta3) for nodeRegistration.Taint to not mention "master" taint and be more generic. Remove example for taints that includes the word "master". - Update unit tests.

- During "upgrade apply" call a new function AddNewControlPlaneTaint() that finds all nodes with the new "control-plane" node-role label and adds the new "control-plane" taint to them. - The function is called in "apply" and is separate from the step to remove the old "master" label for better debugging if errors occur.

neolit123 · 2022-01-13T15:55:12Z

/triage accepted
/priority important-soon
/milestone v1.24

neolit123 · 2022-01-13T15:55:33Z

/approve cancel

pacoxu · 2022-01-14T02:34:45Z

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/107533/pull-kubernetes-e2e-kind/1481653925294641152
failed for taint change.

Jan 13 16:22:25.434: INFO: -> Node kind-control-plane [[[ Ready=true, Network(available)=false, Taints=[{node-role.kubernetes.io/master NoSchedule } {node-role.kubernetes.io/control-plane NoSchedule }], NonblockingTaints=node-role.kubernetes.io/master ]]]

kubernetes/test/e2e/framework/test_context.go

Line 301 in e045e5c

    
           flags.BoolVar(&TestContext.DisableLogDump, "disable-log-dump", false, "If set to true, logs from master and nodes won't be gathered after test run.")

pull-kubernetes-e2e-kind can set non-blocking-taints with flag. Or we should change its default value?

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com> (cherry picked from commit 78b1a796be4ed5b587dbfa5d246702923574250f)

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane WARNING: openshift 4.11 still uses the old label. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com> (cherry picked from commit 1094ec9)

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Partial cherry-pick from 1094ec9 for ovnkube-trace only. Authored-by: Riccardo Ravaioli <rravaiol@redhat.com> Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com> Signed-off-by: Andreas Karis <ak.karis@gmail.com>

As per kubernetes/kubernetes#107533 applied in kubernetes 1.24, replace all occurences of node-role.kubernetes.io/master with node-role.kubernetes.io/control-plane Cherry-pick for ovnkube-trace.go only Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com> (cherry picked from commit 1094ec9) Signed-off-by: Andreas Karis <ak.karis@gmail.com>

The master label is no longer present on kubeadm control plane nodes(v1.24). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to KEP-2067[https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint]: Rename the kubeadm "master" label and taint. the kubernetes pr: kubernetes/kubernetes#107533 Change-Id: I3056b642db0a1799089998e3c020b4203c9a93ab

* Update openstack-helm-infra from branch 'master' to 6034a00bf7049d2a7f10bd946d04436a3041ed87 - Replace node-role.kubernetes.io/master with control-plane The master label is no longer present on kubeadm control plane nodes(v1.24). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to KEP-2067[https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint]: Rename the kubeadm "master" label and taint. the kubernetes pr: kubernetes/kubernetes#107533 Change-Id: I3056b642db0a1799089998e3c020b4203c9a93ab

* Update openstack-helm from branch 'master' to 73e696b3fbbbbcfb3b6acb1b191885f98ff2b18d - Replace node-role.kubernetes.io/master with control-plane The master label is no longer present on kubeadm control plane nodes(v1.24). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to KEP-2067[https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint]: Rename the kubeadm "master" label and taint. the kubernetes pr: kubernetes/kubernetes#107533 Change-Id: Iad8c833371efb3ec35149c89eb8fafdf1150fa87

The master label is no longer present on kubeadm control plane nodes(v1.24). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to KEP-2067[https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint]: Rename the kubeadm "master" label and taint. the kubernetes pr: kubernetes/kubernetes#107533 Change-Id: Iad8c833371efb3ec35149c89eb8fafdf1150fa87

neolit123 added 4 commits January 13, 2022 17:45

k8s-ci-robot requested review from fabriziopandini and pacoxu January 13, 2022 15:46

This was referenced Jan 13, 2022

rename "node-role.kubernetes.io/master" kubernetes/kubeadm#2200

Closed

Rename references to "node-role.kubernetes.io/master" kubernetes-sigs/cluster-api#3279

Closed

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jan 13, 2022

k8s-ci-robot added this to the v1.24 milestone Jan 13, 2022

k8s-ci-robot removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jan 13, 2022

k8s-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 13, 2022

ricky-rav mentioned this pull request Jun 17, 2022

bump k8s to 1.24.0 ovn-org/ovn-kubernetes#3001

Merged

mboersma mentioned this pull request Jun 21, 2022

Switch to "control-plane" labels and taints for v1.20 or afterward Azure/aks-engine#4050

Closed

adikul30 mentioned this pull request Jul 19, 2022

cns-csi.yaml changes for k8s 1.24 kubernetes-sigs/vsphere-csi-driver#1875

Merged

ddubson mentioned this pull request Aug 19, 2022

Benchmark Job on Control Plane does not work with K8s 1.24.x aquasecurity/kube-bench#1250

Closed

adikul30 mentioned this pull request Sep 22, 2022

cns-csi.yaml changes for k8s 1.24 kubernetes-sigs/vsphere-csi-driver#1981

Merged

pwmorreale mentioned this pull request Nov 15, 2022

Fix create_stack for version 1.25 clearlinux/cloud-native-setup#347

Merged

pacoxu mentioned this pull request Jan 12, 2023

Add a flag to toggle removing master taints on control-plane node in kubeadm 1.25 kubernetes/kubeadm#2802

Closed

technerd4357 mentioned this pull request Jan 20, 2023

Rancher UI shows control plane nodes as workers rancher/rancher#40235

Open

fiunchinho mentioned this pull request Mar 16, 2023

Replace deprecated node-role.kubernetes.io/master taint giantswarm/roadmap#2135

Closed

innobead mentioned this pull request Apr 15, 2023

kubeadm bootstrapping, taint "node-role.kubernetes.io/master" not found innobead/kubefire#75

Closed

asifdxtreme mentioned this pull request Apr 20, 2023

vsphere-csi-driver refresh to latest version IBM/ibm-satellite-storage#284

Closed

SataQiu mentioned this pull request Jun 6, 2023

e2e-framework: remove node-role.kubernetes.io/master taint from non-blocking-taints #118510

Merged

fossedihelm mentioned this pull request Apr 18, 2024

Require scheduling infra components onto control-plane nodes kubevirt/kubevirt#11659

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: apply "master" label/taint migration for 1.24 #107533

kubeadm: apply "master" label/taint migration for 1.24 #107533

neolit123 commented Jan 13, 2022 •

edited

Loading

neolit123 commented Jan 13, 2022

neolit123 commented Jan 13, 2022

pacoxu commented Jan 14, 2022 •

edited

Loading

kubeadm: apply "master" label/taint migration for 1.24 #107533

kubeadm: apply "master" label/taint migration for 1.24 #107533

Conversation

neolit123 commented Jan 13, 2022 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

neolit123 commented Jan 13, 2022

neolit123 commented Jan 13, 2022

pacoxu commented Jan 14, 2022 • edited Loading

neolit123 commented Jan 13, 2022 •

edited

Loading

pacoxu commented Jan 14, 2022 •

edited

Loading