Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Merged
merged 1 commit into from Jul 31, 2019

Conversation

@Arvinderpal
Copy link
Contributor

commented Jul 24, 2019

What type of PR is this?
/kind feature

What this PR does / why we need it:

This PR propagates the kubeadm feature-gate, if set in ClusterConfiguration, to the all individual kubernetes components on the control-plane node.

kind: ClusterConfiguration
featureGates:
  IPv6DualStack: true

For worker nodes, users will have to specify the dual-stack feature-gate to kubelet using the nodeRegistration.kubeletExtraArgs option as part of their join config. Alternatively, they can use KUBELET_EXTRA_ARGS as well.

Which issue(s) this PR fixes:

This addresses one of the TODO items in the dual-stack kubeadm tracker ticket: kubernetes/kubeadm#1612
It's a follow up to the initial kubeadm dual-stack feature-gate PR: #80145

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

In order to enable dual-stack support within kubeadm and kubernetes components, as part of the init config file, the user should set feature-gate IPv6DualStack=true in the ClusterConfiguration. Additionally, for each worker node, the user should set the feature-gate for kubelet using either nodeRegistration.kubeletExtraArgs or  KUBELET_EXTRA_ARGS.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jul 24, 2019

Hi @Arvinderpal. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@Arvinderpal

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2019

@neolit123 @yastij @fabriziopandini @aojea
As requested, this PR propagates the dual-stack feature-gate flag to all kubernetes components.
I made this a separate PR from the comma separated list support for --cluster-cidr. I did not want to mix these two together.

@Arvinderpal Arvinderpal referenced this pull request Jul 24, 2019
5 of 9 tasks complete
@neolit123

This comment has been minimized.

Copy link
Member

commented Jul 24, 2019

/assign
/priority important-longterm
/ok-to-test

@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch 2 times, most recently from 0556136 to 86be728 Jul 24, 2019
Copy link
Member

left a comment

thanks @Arvinderpal
added minor comments, but overall this is looking good.

cmd/kubeadm/app/util/config/common.go Outdated Show resolved Hide resolved
@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
}

if isDualStack {

This comment has been minimized.

Copy link
@neolit123

neolit123 Jul 24, 2019

Member

can be:
if features.Enabled(cfg.ClusterConfiguration.FeatureGates, features.IPv6DualStack) {
to avoid passing the isDualStack arg
[1]

This comment has been minimized.

Copy link
@Arvinderpal

Arvinderpal Jul 25, 2019

Author Contributor

That won't work since we pass in cfg *kubeadmapi.NodeRegistrationOptions and not ClusterConfiguration

This comment has been minimized.

Copy link
@neolit123

neolit123 Jul 25, 2019

Member

ah i see, so one solution is to make SetNodeRegistrationDynamicDefaults accept ClusterConfiguration (it think it used to be like that at some point) 🤔

@rosti WDYT?
/cc @rosti

This comment has been minimized.

Copy link
@rosti

rosti Jul 26, 2019

Member

This is not the place to do the feature gate patching for the kubelet. The correct place is buildKubeletArgMap:

func buildKubeletArgMap(opts kubeletFlagsOpts) map[string]string {

This comment has been minimized.

Copy link
@Arvinderpal

Arvinderpal Jul 28, 2019

Author Contributor

@rosti I see a problem with how BuildArgumentListFromMap works today -- it does not handle comma separated fields like "feature-gates", since it does a simple overwrite of default values with user specified values. For example, if we set IPv6DualStack=true in base, then that will be overwritten if the user has specified some additional feature-gates in KubeletExtraArgs.

We have a couple of options:

  1. Make BuildArgumentListFromMap capable of "merging" fields like feature-gates rather than simple key/value overwrite. We would do something like this:
for k, v := range overrideArguments {
		if k == "feature-gates" && len(argsMap[k]) > 0{
			argsMap[k] += "," + v
		}else{
			argsMap[k] = v
		}
	}
  1. Modify top-level funcs like WriteKubeletDynamicEnvFile so that they perform the merge on feature-gates. This would require updating the argList list returned by BuildArgumentListFromMap. This is not ideal, IMO.

We could also just leave the code in the higher-level funcs as it is in the current PR.
I'll wait for you feedback before proceeding. Thanks.

This comment has been minimized.

Copy link
@rosti

rosti Jul 29, 2019

Member

Let's keep things simple for now and just allow the users to overwrite the whole of the feature-gates flags if they wish.
Either way, this is going to be documented as part of the feature gate documentation.
Another possibility is to modify BuildArgumentListFromMap to warn on overwrites.

Certainly, we should not do this in the dynamic defaulting code.

@fabriziopandini @neolit123 WDYT?

This comment has been minimized.

Copy link
@neolit123

neolit123 Jul 29, 2019

Member

i think that @rosti is correct here that dynamic defaults is not the best place.

Let's keep things simple for now and just allow the users to overwrite the whole of the feature-gates flags if they wish.

makes sense to me.

passing the dual stack feature gate to the kubelet will be implicit (if the kubeadm FG is on). so if the user provides their own feature-gates for the kubelet extraArgs, this should overwrite the implicit gate but should also show a warning.

Either way, this is going to be documented as part of the feature gate documentation.

we might add a sentence about overrides, yes.

This comment has been minimized.

Copy link
@Arvinderpal

Arvinderpal Jul 29, 2019

Author Contributor

Done.

cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common_test.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common_test.go Outdated Show resolved Hide resolved
@neolit123

This comment has been minimized.

Copy link
Member

commented Jul 24, 2019

/retest

@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from 86be728 to ec15c65 Jul 25, 2019
Copy link
Contributor Author

left a comment

@neolit123 Thanks for the feedback. Please see my comments.

@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
}

if isDualStack {

This comment has been minimized.

Copy link
@Arvinderpal

Arvinderpal Jul 25, 2019

Author Contributor

That won't work since we pass in cfg *kubeadmapi.NodeRegistrationOptions and not ClusterConfiguration

cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common_test.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/util/config/common_test.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot requested a review from rosti Jul 25, 2019
@aojea

This comment has been minimized.

Copy link
Contributor

commented Jul 25, 2019

/test pull-kubernetes-integration

Copy link
Member

left a comment

Thanks @Arvinderpal !
However, we need the feature gate propagation to be done at a more lower code level, than in the config dynamic defaulting code.

@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
}

if isDualStack {

This comment has been minimized.

Copy link
@rosti

rosti Jul 26, 2019

Member

This is not the place to do the feature gate patching for the kubelet. The correct place is buildKubeletArgMap:

func buildKubeletArgMap(opts kubeletFlagsOpts) map[string]string {

cmd/kubeadm/app/util/config/initconfiguration.go Outdated Show resolved Hide resolved
@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from ec15c65 to 56ad9b6 Jul 29, 2019
@k8s-ci-robot k8s-ci-robot added size/M and removed size/L labels Jul 29, 2019
@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from 56ad9b6 to 6a58483 Jul 29, 2019
@Arvinderpal

This comment has been minimized.

Copy link
Contributor Author

commented Jul 29, 2019

@rosti @neolit123 Thank you for the feedback. I have made the changes requested.
It actually makes things a lot simpler now :)

@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from 6a58483 to 6082e7f Jul 29, 2019
Copy link
Member

left a comment

thanks for the update
/lgtm
approve on @rosti

@k8s-ci-robot k8s-ci-robot added the lgtm label Jul 29, 2019
@neolit123

This comment has been minimized.

Copy link
Member

commented Jul 29, 2019

looks like some build tests are failing.

@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from 6082e7f to bec0017 Jul 30, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm label Jul 30, 2019
@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from bec0017 to 6524f81 Jul 30, 2019
Copy link
Member

left a comment

just some nits, also +1 on the previous feedback @rosti

enabled, present := cfg.FeatureGates[features.IPv6DualStack]
if present && enabled {
defaultArguments["feature-gates"] = features.IPv6DualStack + "=true"
} else if present && !enabled {

This comment has been minimized.

Copy link
@yastij

yastij Jul 30, 2019

Member

why not just else ? (the remaining case is !present)

This comment has been minimized.

Copy link
@rosti

rosti Jul 30, 2019

Member

[1] Ideally, we would like to cover the present==true case. So something like this should work:

if present {
        defaultArguments["feature-gates"] = fmt.Sprintf("%s=%t", features.IPv6DualStack, enabled)
}

This comment has been minimized.

Copy link
@Arvinderpal

Arvinderpal Jul 30, 2019

Author Contributor

Done!

cmd/kubeadm/app/phases/controlplane/manifests.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/phases/kubelet/flags.go Outdated Show resolved Hide resolved
Copy link
Member

left a comment

Thanks @Arvinderpal !
Leaving final lgtm to @yastij or @neolit123
/approve

enabled, present := cfg.FeatureGates[features.IPv6DualStack]
if present && enabled {
defaultArguments["feature-gates"] = features.IPv6DualStack + "=true"
} else if present && !enabled {

This comment has been minimized.

Copy link
@rosti

rosti Jul 30, 2019

Member

[1] Ideally, we would like to cover the present==true case. So something like this should work:

if present {
        defaultArguments["feature-gates"] = fmt.Sprintf("%s=%t", features.IPv6DualStack, enabled)
}
cmd/kubeadm/app/phases/controlplane/manifests.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/phases/controlplane/manifests.go Outdated Show resolved Hide resolved
cmd/kubeadm/app/phases/kubelet/flags.go Outdated Show resolved Hide resolved
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Arvinderpal, rosti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

progagates the kubeadm FG to the individual k8scomponents
on the control-plane node.

* Note: Users who want to join worker nodes to the cluster
will have to specify the dual-stack FG to kubelet using the
nodeRegistration.kubeletExtraArgs option as part of their
join config. Alternatively, they can use KUBELET_EXTRA_ARGS.

kubeadm FG: kubernetes/kubeadm#1612
@Arvinderpal Arvinderpal force-pushed the Nordix:kubeadm-ds-FG-propagate branch from 6524f81 to 585ef37 Jul 30, 2019
@yastij

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

/lgtm

@Arvinderpal

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

/retest

@fejta-bot

This comment has been minimized.

Copy link

commented Jul 30, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

2 similar comments
@fejta-bot

This comment has been minimized.

Copy link

commented Jul 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

commented Jul 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 5bfa366 into kubernetes:master Jul 31, 2019
20 of 23 checks passed
20 of 23 checks passed
pull-kubernetes-e2e-gce Job triggered.
Details
pull-kubernetes-integration Job triggered.
Details
pull-kubernetes-kubemark-e2e-gce-big Job triggered.
Details
cla/linuxfoundation Arvinderpal authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot added this to the v1.16 milestone Jul 31, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

@Arvinderpal: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce 585ef37 link /test pull-kubernetes-e2e-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.