🌱 Support new control plane label and taint #5919

sbueringer · 2022-01-10T14:01:48Z

Signed-off-by: Stefan Büringer buringerst@vmware.com

What this PR does / why we need it:
This PR is required to be compatible with the latest changes in Kubernetes 1.24.

In Kubernetes 1.24 on control plane nodes kubeadm now:

sets both node-role.kubernetes.io/control-plane and node-role.kubernetes.io/master taints
doesn't set the node-role.kubernetes.io/master label anymore (only node-role.kubernetes.io/control-plane)

Thus:

the toleration node-role.kubernetes.io/control-plane is additionally added so our controllers can also run on 1.24 control-plane nodes.
KCP now considers nodes with the old or the new (or both) label(s) as control plane nodes. This way KCP now supports:
- v1.19: where only the old label is set
- v1.20-v1.23: where both labels are set
- v1.24: where only the new label is set

xref: corresponding kubeadm issue kubernetes/kubeadm#2200

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Partially implements #3279

controlplane/kubeadm/internal/workload_cluster.go

vincepri

Minor nit, rest LGTM

controlplane/kubeadm/internal/workload_cluster.go

sbueringer · 2022-01-10T15:59:27Z

controlplane/kubeadm/internal/workload_cluster_conditions_test.go

@@ -980,6 +980,9 @@ func fakeNode(name string, options ...fakeNodeOption) *corev1.Node {
 	p := &corev1.Node{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: name,
+			Labels: map[string]string{
+				labelNodeRoleControlPlane: "",


This is now necessary because the old implementation of getControlPlaneNodes did select all nodes as control plane nodes independent of the label when using fakeClient (I assume because ctrlclient.MatchingLabels(labels) is not implemented in fake client)

There are tests for it https://github.com/kubernetes-sigs/controller-runtime/blob/f20692b7ce2a9dda7eb298504a9c47aaf5a96544/pkg/client/fake/client_test.go#L314-L324 and it should be supported afaik.

The code that does the selection is in https://github.com/kubernetes-sigs/controller-runtime/blob/4e7f0c968b90a306fb9ef22f8e92fa36b98b4ef1/pkg/cache/internal/cache_reader.go#L107-L108 — are we not using a cache here perhaps?

I'll take a closer look.

To be clear for the new code everything works fine. I think it's now mostly a question of why didn't we have to set the control plane node label with the old code. (aka is there something wrong with fake client or our usage of it)

Okay everything worked/works as expected.

In the relevant unit test we're injecting the list result:

cluster-api/controlplane/kubeadm/internal/workload_cluster_conditions_test.go

Lines 84 to 88 in 5026786

injectClient: &fakeClient{

list: &corev1.NodeList{

Items: []corev1.Node{*fakeNode("n1")},

},

},

Previously, we didn't have to set the control plane label as the result of the list was automatically the list of control plane nodes. As we're now iterating through the result and checking for the label it became relevant.

k8s-ci-robot · 2022-01-10T16:03:10Z

@sbueringer: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main
/test pull-cluster-api-e2e-main
/test pull-cluster-api-test-main
/test pull-cluster-api-test-mink8s-main
/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main
/test pull-cluster-api-e2e-full-main
/test pull-cluster-api-e2e-informing-ipv6-main
/test pull-cluster-api-e2e-informing-main
/test pull-cluster-api-e2e-workload-upgrade-1-23-latest-main
/test pull-cluster-api-make-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main
pull-cluster-api-build-main
pull-cluster-api-e2e-informing-ipv6-main
pull-cluster-api-e2e-informing-main
pull-cluster-api-e2e-main
pull-cluster-api-test-main
pull-cluster-api-test-mink8s-main
pull-cluster-api-verify-main

In response to this:

/test
pull-cluster-api-verify-main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2022-01-10T16:03:19Z

/test pull-cluster-api-verify-main

enxebre · 2022-01-10T16:07:06Z

thanks @sbueringer!
/lgtm

CecileRobertMichon · 2022-01-10T18:46:12Z

test/framework/deployment_helpers.go

+
+		// Use the control-plane label for Kubernetes version >= v1.20.0.
+		if utilversion.MustParseGeneric(serverVersion.String()).AtLeast(utilversion.MustParseGeneric("v1.20.0")) {
+			workloadDeployment.Spec.Template.Spec.NodeSelector = map[string]string{"node-role.kubernetes.io/control-plane": ""}


should the strings be constants here and below?

enxebre · 2022-01-11T15:22:44Z

/lgtm

neolit123 · 2022-01-13T15:50:42Z

found this entry in the TS guide that might have to be updated:
https://github.com/kubernetes-sigs/cluster-api/blob/4860222a821b02011a3a2c0051e62332837764cb/docs/book/src/user/troubleshooting.md#labeling-nodes-with-reserved-labels-such-as-node-rolekubernetesio-fails-with-kubeadm-error-during-bootstrap

neolit123 · 2022-01-13T15:53:22Z

In Kubernetes 1.24 on control plane nodes kubeadm now:

uses the taint node-role.kubernetes.io/control-plane instead of node-role.kubernetes.io/master

both will be applied on nodes in 1.24. xref KEP:
https://github.com/neolit123/enhancements/tree/c33232cf66cc911ec736fcc80a08bd42189f8504/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint

doesn't set the node-role.kubernetes.io/master label anymore (only node-role.kubernetes.io/control-plane)

yes, for new CP nodes (init or join) only the "control-plane" label will be added.

sbueringer · 2022-01-13T16:05:49Z

Thank you! Will update accordingly

sbueringer · 2022-01-17T13:41:49Z

I've updated getControlPlaneNodes to use continue, so we're not missing any control plane nodes in large clusters.

@fabriziopandini ptal :)

sbueringer · 2022-02-01T15:44:04Z

Hey folks,
WDYT about merging this PR?

controlplane/kubeadm/internal/workload_cluster.go

Signed-off-by: Stefan Büringer buringerst@vmware.com

vincepri

/approve
/lgtm

vincepri · 2022-02-01T20:28:11Z

Should we include this PR in the v1.1 release?

k8s-ci-robot · 2022-02-01T20:28:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2022-02-01T20:54:19Z

Should we include this PR in the v1.1 release?

I'm not sure. On one side we would need this PR to support v1.24, on the other side we will need more than that. (see release specific issues in #5968)

So this PR alone is not enough to support the upcoming v1.24 release.

I'm also not sure if we want to use v1.24 support as a forcing function so folks are upgrading to CAPI v1.2 :)

Even if we don't cherry-pick now, once we know what we need to support v1.24.0 (after its release) we can still cherry-pick that for a new v1.1.x release if we have consensus that we want to cherry-pick 1.24 support into CAPI v1.1.x

/cc @fabriziopandini

enxebre · 2022-02-02T09:13:03Z

So this PR alone is not enough to support the upcoming v1.24 release.

fwiw 1.24 is planned for Tuesday 19th April 2022

Given the timelines I'd be in favour of supporting 1.24 in 1.1.x.
My +1 to cherry-pick now into 1.1 just so I'd rather cherry-pick smaller bits than a bigger one in one go so we keep the potential breaking surface controlled.

For ref #5968

sbueringer · 2022-02-02T09:14:31Z

I don't have a strong opinion either way, but we should only consider cherry-picking it if we plan to support v1.24 in v1.1.x

sbueringer · 2022-02-09T13:35:45Z

We'll bring it up in the office hours today, but we would want to support v1.24 in v1.1.x. (we'll wait with merging the cherry-pick until after the meeting)
/cherry-pick release-1.1

k8s-infra-cherrypick-robot · 2022-02-09T13:36:21Z

@sbueringer: new pull request created: #6084

In response to this:

We'll bring it up in the office hours today, but we would want to support v1.24 in v1.1.x. (we'll wait with merging the cherry-pick)
/cherry-pick release-1.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kashifest · 2022-07-04T06:53:16Z

@sbueringer Would it be ok to cherry pick this for release-0.4 ? If not, v1a4 cluster wont be able to migrate to 1.24.x because of the missing node role label check.

sbueringer · 2022-07-04T07:00:15Z

CAPI v0.4.x doesn't support Kubernetes 1.24: https://cluster-api.sigs.k8s.io/reference/versions.html

For v1.24 support you have to upgrade to CAPI v1.1.

Note: CAPI v0.4 is out of support since 2022-04-06 and there won't be another v0.4.x release.

P.S. This is not the only PR we would have to backport to make this happen

kashifest · 2022-07-04T07:01:28Z

CAPI v0.4.x doesn't support Kubernetes 1.24: https://cluster-api.sigs.k8s.io/reference/versions.html

For v1.24 support you have to upgrade to CAPI v1.1.

Note: CAPI v0.4 is out of support since 2022-04-06 and there won't be another v0.4.x release.

P.S. This is not the only PR we would have to backport to make this happen

Ah ok , missed that, thanks

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 10, 2022

k8s-ci-robot requested review from CecileRobertMichon and JoelSpeed January 10, 2022 14:02

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 10, 2022

enxebre reviewed Jan 10, 2022

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Show resolved Hide resolved

vincepri reviewed Jan 10, 2022

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Outdated Show resolved Hide resolved

sbueringer force-pushed the pr-controlplane-label-taint branch from 1982b7a to 892bd88 Compare January 10, 2022 15:58

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 10, 2022

sbueringer commented Jan 10, 2022

View reviewed changes

k8s-ci-robot assigned enxebre Jan 10, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 10, 2022

CecileRobertMichon reviewed Jan 10, 2022

View reviewed changes

sbueringer force-pushed the pr-controlplane-label-taint branch from 892bd88 to 6893268 Compare January 11, 2022 14:59

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2022

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 13, 2022

sbueringer force-pushed the pr-controlplane-label-taint branch from 6893268 to a758f8f Compare January 13, 2022 16:12

k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jan 13, 2022

sbueringer force-pushed the pr-controlplane-label-taint branch from a758f8f to c2a843c Compare January 17, 2022 13:39

sbueringer force-pushed the pr-controlplane-label-taint branch from c2a843c to eb6d6b6 Compare January 17, 2022 13:54

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 26, 2022

vincepri reviewed Feb 1, 2022

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Show resolved Hide resolved

sbueringer force-pushed the pr-controlplane-label-taint branch from eb6d6b6 to f44f816 Compare February 1, 2022 19:36

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 1, 2022

Support new control plane label and taint

2497fcc

Signed-off-by: Stefan Büringer buringerst@vmware.com

sbueringer force-pushed the pr-controlplane-label-taint branch from f44f816 to 2497fcc Compare February 1, 2022 19:41

vincepri approved these changes Feb 1, 2022

View reviewed changes

k8s-ci-robot assigned vincepri Feb 1, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 1, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 1, 2022

k8s-ci-robot merged commit 5756be1 into kubernetes-sigs:main Feb 1, 2022

k8s-ci-robot added this to the v1.2 milestone Feb 1, 2022

k8s-ci-robot requested a review from fabriziopandini February 1, 2022 20:54

k8s-infra-cherrypick-robot mentioned this pull request Feb 9, 2022

🌱 Support new control plane label and taint #6084

Merged

sbueringer deleted the pr-controlplane-label-taint branch February 9, 2022 13:37

sbueringer mentioned this pull request Feb 11, 2022

Tasks for Kubernetes v1.24 #5968

Closed

33 tasks

CecileRobertMichon mentioned this pull request Feb 17, 2022

Add control-plane role taints and tolerations kubernetes-sigs/cluster-api-provider-azure#2098

Merged

3 tasks

chrischdi mentioned this pull request Apr 19, 2022

🐛 bump CAPI to v1.1.3 kubernetes-sigs/cluster-api-provider-openstack#1209

Merged

3 tasks

jichenjc mentioned this pull request Apr 20, 2022

[All] update node-role.kubernetes.io/master=>node-role.kubernetes.io/control-plane kubernetes/cloud-provider-openstack#1839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Support new control plane label and taint #5919

🌱 Support new control plane label and taint #5919

sbueringer commented Jan 10, 2022 •

edited

vincepri left a comment

sbueringer Jan 10, 2022 •

edited

vincepri Jan 18, 2022

sbueringer Jan 19, 2022 •

edited

sbueringer Jan 19, 2022

k8s-ci-robot commented Jan 10, 2022

sbueringer commented Jan 10, 2022

enxebre commented Jan 10, 2022

CecileRobertMichon Jan 10, 2022

sbueringer Jan 11, 2022

enxebre commented Jan 11, 2022

neolit123 commented Jan 13, 2022

neolit123 commented Jan 13, 2022

sbueringer commented Jan 13, 2022

sbueringer commented Jan 17, 2022

sbueringer commented Feb 1, 2022

vincepri left a comment

vincepri commented Feb 1, 2022

k8s-ci-robot commented Feb 1, 2022

sbueringer commented Feb 1, 2022 •

edited

enxebre commented Feb 2, 2022 •

edited

sbueringer commented Feb 2, 2022

sbueringer commented Feb 9, 2022 •

edited

k8s-infra-cherrypick-robot commented Feb 9, 2022

kashifest commented Jul 4, 2022

sbueringer commented Jul 4, 2022 •

edited

kashifest commented Jul 4, 2022

	injectClient: &fakeClient{
	list: &corev1.NodeList{
	Items: []corev1.Node{*fakeNode("n1")},
	},
	},

🌱 Support new control plane label and taint #5919

🌱 Support new control plane label and taint #5919

Conversation

sbueringer commented Jan 10, 2022 • edited

vincepri left a comment

Choose a reason for hiding this comment

sbueringer Jan 10, 2022 • edited

Choose a reason for hiding this comment

vincepri Jan 18, 2022

Choose a reason for hiding this comment

sbueringer Jan 19, 2022 • edited

Choose a reason for hiding this comment

sbueringer Jan 19, 2022

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 10, 2022

sbueringer commented Jan 10, 2022

enxebre commented Jan 10, 2022

CecileRobertMichon Jan 10, 2022

Choose a reason for hiding this comment

sbueringer Jan 11, 2022

Choose a reason for hiding this comment

enxebre commented Jan 11, 2022

neolit123 commented Jan 13, 2022

neolit123 commented Jan 13, 2022

sbueringer commented Jan 13, 2022

sbueringer commented Jan 17, 2022

sbueringer commented Feb 1, 2022

vincepri left a comment

Choose a reason for hiding this comment

vincepri commented Feb 1, 2022

k8s-ci-robot commented Feb 1, 2022

sbueringer commented Feb 1, 2022 • edited

enxebre commented Feb 2, 2022 • edited

sbueringer commented Feb 2, 2022

sbueringer commented Feb 9, 2022 • edited

k8s-infra-cherrypick-robot commented Feb 9, 2022

kashifest commented Jul 4, 2022

sbueringer commented Jul 4, 2022 • edited

kashifest commented Jul 4, 2022

sbueringer commented Jan 10, 2022 •

edited

sbueringer Jan 10, 2022 •

edited

sbueringer Jan 19, 2022 •

edited

sbueringer commented Feb 1, 2022 •

edited

enxebre commented Feb 2, 2022 •

edited

sbueringer commented Feb 9, 2022 •

edited

sbueringer commented Jul 4, 2022 •

edited