kubeadm: Split discovery from JoinConfiguration #67763

rosti · 2018-08-23T11:20:07Z

What this PR does / why we need it:

This change splits out discovery fields from JoinConfiguration by performing
the following changes:

Introduce a BootstrapTokenDiscovery structure, that houses configuration
options needed for bootstrap token based discovery.
Introduce a FileDiscovery structure, that houses configuration options
(currently only a single option) needed for KubeConfig based discovery.
Introduce a Discovery structure, that houses common options (such as
discovery timeout and TLS bootstrap token) as well as pointer to an instance
of either BootstrapTokenDiscovery or FileDiscovery structures.
Replace the old discovery related JoinConfiguration members with a single
Discovery member.

This change is required in order to cleanup the code of unnecessary logic and
make the serialized JoinConfiguration more structured (and therefore, more
intuitive).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
refs kubernetes/kubeadm#911, refs kubernetes/kubeadm#963

Special notes for your reviewer:

This is another one of those big and somewhat hard to review PRs, that are the stepping stones in the path to v1beta1.

/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/kind api-change
/kind enhancement
/assign @luxas
/assign @timothysc
/assign @fabriziopandini

Release note:

kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery)

dixudx

For current v1alpha3, I'd prefer not to update the json structure. Use inline instead.

We could update that in next coming v1beta1.

dixudx · 2018-08-23T11:33:27Z

cmd/kubeadm/app/cmd/join.go

+
+			if len(fd.KubeConfigPath) != 0 {
+				cfg.Discovery.File = fd
+				cfg.Discovery.BootstrapToken = nil


Default will be nil.
Remove it.

dixudx · 2018-08-23T11:33:36Z

cmd/kubeadm/app/cmd/join.go

+				cfg.Discovery.BootstrapToken = nil
+			} else {
+				cfg.Discovery.BootstrapToken = btd
+				cfg.Discovery.File = nil


dixudx · 2018-08-23T11:38:39Z

cmd/kubeadm/app/util/config/testdata/validation/invalid_nodecfg.yaml

 nodeRegistration:
  criSocket: relativepath
-  name: NODE-1
-token: invalidtoken
+  name: NODE-1


Missing newline?

dixudx · 2018-08-23T11:39:10Z

cmd/kubeadm/app/util/config/testdata/conversion/node/internal.yaml

- kube-apiserver:6443
-DiscoveryTokenCACertHashes: null
-DiscoveryTokenUnsafeSkipCAVerification: true
+Discovery:


This will break old configurations.

For coming v1beta1, we cound use a new section like this.

But for current v1alpha3, shall we use below instead in types.go,

Discovery Discovery `json:",inline"`

Actually, it won't, since v1alpha3 is also unreleased and is massively different than v1alpha2 of 1.11.
I am also not a fan of inlining it, because that way the context of the options in Discovery is gone.

dixudx · 2018-08-23T11:39:18Z

cmd/kubeadm/app/util/config/testdata/conversion/node/v1alpha3.yaml

-discoveryTokenAPIServers:
- kube-apiserver:6443
-discoveryTokenUnsafeSkipCAVerification: true
+discovery:


Ditto for those following yaml files.

Better not change the struction of current released v1alpha3. We could do this in coming v1beta1 instead.

rosti · 2018-08-23T12:33:28Z

For current v1alpha3, I'd prefer not to update the json structure. Use inline instead.

We could update that in next coming v1beta1.

This looks like a double work for me and I am willing to actually hold of this PR for v1beta1 if we are going to split it into user and back facing interface changes.
I am not a big fan of inlining the Discovery structure either - it gives us more context that way and makes the config file more structured.
The fact is that this change is literally tiny compared to the rest if the changes we have done in v1alpha3. I don't believe that a user facing interface change here is going to break anyone, since v1alpha3 is unreleased.

timothysc

/hold

please coordinate this with @fabriziopandini there are overlapping issues right now.

rosti · 2018-10-08T14:09:47Z

Rebased the PR on v1beta1

@fabriziopandini @timothysc PTAL

timothysc · 2018-10-09T21:39:22Z

@rosti bot says it still needs a rebase.
I'd like to chat about config changes tomorrow during office hours too.

timothysc

Given that you and @fabriziopandini have been working on this one I'll defer to him on lgtm

/approve
/assign @fabriziopandini

fabriziopandini

@rosti thank you for this PR and +100 for keeping this going across two cycles!

Overall this PR looks fine, but the things that confused a little bit me is the move from tree token fields (discoveryToken, tlsBootstrapToken and token to override the former two) to two token fields. Some other minors inline

fabriziopandini · 2018-10-11T16:24:50Z

cmd/kubeadm/app/apis/kubeadm/types.go

 	// TLSBootstrapToken is a token used for TLS bootstrapping.
-	// Defaults to Token.
+	// If .BootstrapToken is set, this field is defaulted to .BootstrapToken.Token, but can be overridden.
+	// If .File is set, this field **must be set** in case the KubeConfigFile does not contain any other authentication information


Wandering if it makes more sense to have two TLS bootstrap tokens, one inside BootstrapToken and one inside File.
WDYT?

Ok according to original Lucas design

fabriziopandini · 2018-10-11T16:36:46Z

cmd/kubeadm/app/apis/kubeadm/types.go

+// FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information
+type FileDiscovery struct {
+	// KubeConfigPath is used to specify the actual file path or URL to the kubeconfig file from which to load cluster information
+	KubeConfigPath string


I'm not fully convinced by calling this path, but I don't have better suggestion ATM...

How about KubeConfigFile? Yet it sounds kind of the same to me.

Leave as it is ATM

fabriziopandini · 2018-10-11T16:43:54Z

cmd/kubeadm/app/apis/kubeadm/types.go

+}
+
+// BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery
+type BootstrapTokenDiscovery struct {


If I'm not wrong we are missing DiscoveryToken (unless it is intentional to conflate Token and DiscoveryToken)

Ok according to original Lucas design

fabriziopandini · 2018-10-11T16:47:03Z

cmd/kubeadm/app/apis/kubeadm/v1alpha3/conversion.go

+	return nil
+}
+
+func Convert_kubeadm_JoinConfiguration_To_v1alpha3_JoinConfiguration(in *kubeadm.JoinConfiguration, out *JoinConfiguration, s conversion.Scope) error {


if I'm not wrong out.token is missing...

ok according to original Lucas design/the semantic of --token acting as a proxy for TLSBootstrapToken/DiscoveryToken

fabriziopandini · 2018-10-11T16:52:03Z

cmd/kubeadm/app/apis/kubeadm/v1beta1/defaults.go

@@ -152,6 +136,34 @@ func SetDefaults_NodeRegistrationOptions(obj *NodeRegistrationOptions) {
 	}
 }

+// SetDefaults_Discovery assigns default values for the discovery process
+func SetDefaults_Discovery(obj *Discovery) {


DiscoveryToken is missing also here

ok according to original Lucas design

fabriziopandini · 2018-10-11T16:56:33Z

cmd/kubeadm/app/apis/kubeadm/types.go

+	// APIServers is a set of IPs to API servers from which info
+	// will be fetched. Currently we only pay attention to one API server but
+	// hope to support >1 in the future.
+	APIServerEndpoints []string


Wandering if we should make this string instead of accepting array of string and throwing an error in validation if more than one string is passed

Please make this string; we are not accepting more than one value ATM and there is no plan for supporting >1 API Endpoints as far as I know

This will come in as a followup PR as the change is not trivial (currently there is a lot of code in kubeadm, that looks upon this as a []string and doing correct handling of multiple API Servers).

fabriziopandini · 2018-10-11T17:11:17Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

+}
+
+// ValidateFileDiscovery validates file discovery configuration
+func ValidateFileDiscovery(f *kubeadm.FileDiscovery, fldPath *field.Path) field.ErrorList {


might be ValidateDiscoveryFile?

Please rename to ValidateDiscoveryFile

fabriziopandini · 2018-10-11T17:13:06Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

+	}
+
+	allErrs = append(allErrs, ValidateToken(b.Token, fldPath.Child("token"))...)
+	allErrs = append(allErrs, ValidateJoinDiscoveryTokenAPIServer(b.APIServerEndpoints, fldPath.Child("apiServerEndpoints"))...)


this is out of the scope of this PR, but it would be great to rename also ValidateJoinDiscoveryTokenAPIServer into ValidateDiscoveryTokenAPIServer

kindly rename ValidateJoinDiscoveryTokenAPIServer into ValidateDiscoveryTokenAPIServerfor consistency

fabriziopandini · 2018-10-11T17:52:45Z

cmd/kubeadm/app/cmd/join.go

@@ -159,6 +159,10 @@ func NewCmdJoin(out io.Writer) *cobra.Command {
 	cfg := &kubeadmapiv1beta1.JoinConfiguration{}
 	kubeadmscheme.Scheme.Default(cfg)

+	fd := &kubeadmapiv1beta1.FileDiscovery{}


I think to understand why you are creating separated object instead of working on cfg, but I'm wondering if we can do the the other way around: use cfg and then clean up cfg.Discovery.File or cfg.Discovery.BootstrapToken if corresponding flags/args are not set

I'm not a fan of this solution but we can leave with it ATM

fabriziopandini · 2018-10-11T17:57:19Z

cmd/kubeadm/app/cmd/join.go

-		"For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.")
-	flagSet.StringVar(
-		&cfg.Token, "token", "",
+		token, "token", "",


This is a little bit confusing.
In the config we now there is BootstrapToken.Token and TLSBootstrapToken but not Token
here there is Token, TLSBootstrapToken but not discovery-token anymore

Please fix the flag documentation explaining the semantics of this flag

rosti · 2018-10-12T11:22:56Z

@fabriziopandini the change is along the lines of the original proposal by Lucas, which I find the most sensible solution here.
The TLSBootstrapToken is moved to the common Discovery struct and is therefore shared by both discovery methods. The DiscoveryToken field is replaced by BootstrapToken.Token. The Token field, whose purpose in v1alpha3 was to supply a common value to use for TLSBootstrapToken and DiscoveryToken, when no value was supplied for some of them, was removed. You can still specify the --token switch on command line to supply a single value to use for both TLSBootstrapToken and DiscoveryToken. Also, v1alpha3 to internal config is making sure to do the correct conversion too.

P.S.: On second though, I got the semantics for --token a bit different than what it used to be, so I'll change fix them in a bit. They should not be overwritten by --token if they are not empty.

rosti · 2018-10-12T11:29:20Z

Fixed the --token flag semantics (of kubeadm join).

fabriziopandini

@rosti
/approve
Please fix minors still pending and then ready for lgtm

After your explanation I added a note on all my previous comments that does not apply anymore, and make it clear where IMO you should fix minors

fabriziopandini · 2018-10-14T06:44:05Z

cmd/kubeadm/app/apis/kubeadm/types.go

+	// APIServers is a set of IPs to API servers from which info
+	// will be fetched. Currently we only pay attention to one API server but
+	// hope to support >1 in the future.
+	APIServerEndpoints []string


Please make this string; we are not accepting more than one value ATM and there is no plan for supporting >1 API Endpoints as far as I know

fabriziopandini · 2018-10-14T06:44:32Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

-// ValidateArgSelection validates discovery related configuration and collects all encountered errors
-func ValidateArgSelection(cfg *kubeadm.JoinConfiguration, fldPath *field.Path) field.ErrorList {
+// ValidateBootstrapTokenDiscovery validates bootstrap token discovery configuration
+func ValidateBootstrapTokenDiscovery(b *kubeadm.BootstrapTokenDiscovery, fldPath *field.Path) field.ErrorList {


Please rename to ValidateDiscoveryBootstrapToken

fabriziopandini · 2018-10-14T06:44:39Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

 	}

 	// TODO remove once we support multiple api servers
-	if len(cfg.DiscoveryTokenAPIServers) > 1 {
+	if len(b.APIServerEndpoints) > 1 {


This should go away when APIServerEndpoints changes from []string to string

fabriziopandini · 2018-10-14T06:44:44Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

+	}
+
+	allErrs = append(allErrs, ValidateToken(b.Token, fldPath.Child("token"))...)
+	allErrs = append(allErrs, ValidateJoinDiscoveryTokenAPIServer(b.APIServerEndpoints, fldPath.Child("apiServerEndpoints"))...)


kindly rename ValidateJoinDiscoveryTokenAPIServer into ValidateDiscoveryTokenAPIServerfor consistency

fabriziopandini · 2018-10-14T06:44:51Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

+}
+
+// ValidateFileDiscovery validates file discovery configuration
+func ValidateFileDiscovery(f *kubeadm.FileDiscovery, fldPath *field.Path) field.ErrorList {


Please rename to ValidateDiscoveryFile

fabriziopandini · 2018-10-14T06:45:28Z

cmd/kubeadm/app/cmd/join.go

-		"For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.")
-	flagSet.StringVar(
-		&cfg.Token, "token", "",
+		token, "token", "",


Please fix the flag documentation explaining the semantics of this flag

fabriziopandini · 2018-10-16T10:39:04Z

/approve
/lgtm

/test pull-kubernetes-e2e-gce

k8s-ci-robot · 2018-10-16T10:39:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini, rosti, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [fabriziopandini,timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rosti · 2018-10-16T10:56:39Z

A few minor details to address in a couple of hours and this will be ready to go.

/hold

This change splits out discovery fields from JoinConfiguration by performing the following changes: - Introduce a BootstrapTokenDiscovery structure, that houses configuration options needed for bootstrap token based discovery. - Introduce a FileDiscovery structure, that houses configuration options (currently only a single option) needed for KubeConfig based discovery. - Introduce a Discovery structure, that houses common options (such as discovery timeout and TLS bootstrap token) as well as pointer to an instance of either BootstrapTokenDiscovery or FileDiscovery structures. - Replace the old discovery related JoinConfiguration members with a single Discovery member. This change is required in order to cleanup the code of unnecessary logic and make the serialized JoinConfiguration more structured (and therefore, more intuitive). Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>

rosti · 2018-10-16T12:26:55Z

@fabriziopandini should be ready now for re-lgtm.

/hold cancel

fabriziopandini · 2018-10-16T13:05:21Z

/lgtm

fabriziopandini · 2018-10-16T15:54:03Z

/test pull-kubernetes-kubemark-e2e-gce-big

fejta-bot · 2018-10-16T18:10:59Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-10-17T00:07:23Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 23, 2018

k8s-ci-robot assigned fabriziopandini, luxas and timothysc Aug 23, 2018

dixudx suggested changes Aug 23, 2018

View reviewed changes

rosti force-pushed the join-discovery-split branch from 7160887 to 15a5136 Compare August 23, 2018 15:19

timothysc reviewed Aug 24, 2018

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 24, 2018

rosti force-pushed the join-discovery-split branch from 15a5136 to 46777a2 Compare August 27, 2018 10:47

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 27, 2018

rosti force-pushed the join-discovery-split branch from 46777a2 to 7287c86 Compare September 3, 2018 15:03

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 3, 2018

timothysc added this to the 1.13 milestone Sep 5, 2018

timothysc removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 5, 2018

liggitt modified the milestones: 1.13, v1.13 Sep 5, 2018

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 28, 2018

rosti force-pushed the join-discovery-split branch from 7287c86 to 99aa771 Compare October 8, 2018 14:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 8, 2018

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2018

rosti force-pushed the join-discovery-split branch from 99aa771 to 58754a6 Compare October 10, 2018 10:41

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 10, 2018

timothysc reviewed Oct 11, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 11, 2018

fabriziopandini reviewed Oct 11, 2018

View reviewed changes

rosti force-pushed the join-discovery-split branch from 58754a6 to 3ecb446 Compare October 12, 2018 11:28

fabriziopandini suggested changes Oct 14, 2018

View reviewed changes

rosti force-pushed the join-discovery-split branch from 3ecb446 to f41d131 Compare October 15, 2018 15:15

rosti mentioned this pull request Oct 15, 2018

kubeadm: Remove multiple API server endpoints support upon join #69812

Merged

1 task

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2018

rosti force-pushed the join-discovery-split branch from f41d131 to 80971c9 Compare October 16, 2018 12:21

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2018

rosti force-pushed the join-discovery-split branch from 80971c9 to 576b8d3 Compare October 16, 2018 12:25

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2018

k8s-ci-robot merged commit 1e4ad04 into kubernetes:master Oct 17, 2018

rosti deleted the join-discovery-split branch November 22, 2018 15:26

kubeadm: Split discovery from JoinConfiguration #67763

kubeadm: Split discovery from JoinConfiguration #67763

Conversation

rosti commented Aug 23, 2018

dixudx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Aug 23, 2018

timothysc left a comment

Choose a reason for hiding this comment

rosti commented Oct 8, 2018

timothysc commented Oct 9, 2018

timothysc left a comment

Choose a reason for hiding this comment

fabriziopandini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Oct 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Oct 12, 2018

rosti commented Oct 12, 2018

fabriziopandini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini commented Oct 16, 2018

k8s-ci-robot commented Oct 16, 2018

rosti commented Oct 16, 2018

rosti commented Oct 16, 2018

fabriziopandini commented Oct 16, 2018

fabriziopandini commented Oct 16, 2018

fejta-bot commented Oct 16, 2018

fejta-bot commented Oct 17, 2018

fabriziopandini Oct 14, 2018 •

edited

Loading