kubeadm: Controllable timeout for join failures #60983

rosti · 2018-03-09T15:29:20Z

What this PR does / why we need it:

This PR introduces a timeout for kubeadm join. During that time kubeadm will try to join as many times as possible. The timeout can be controlled via the discoveryTimeout config option. Its default value is 5 minutes.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/kubeadm#677

Special notes for your reviewer:

/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/assign @luxas
/assign @timothysc

Release note:

kubeadm: Introduce join timeout that can be controlled via the discoveryTimeout config option (set to 5 minutes by default).

k8s-ci-robot · 2018-03-09T15:29:30Z

@rosti: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cluster-lifecycle-pr-reviews

In response to this:

What this PR does / why we need it:

This PR introduces a 5 minute timeout for kubeadm join. During that time kubeadm will try to join as many times as possible (currently at 5 second intervals, thus having 60 retries).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/kubeadm#677

Special notes for your reviewer:

/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/assign @luxas
/assign @timothysc

Release note:
kubeadm: Introduce join timeout (currently 5 minutes) instead of waiting indefinitely.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2018-03-10T00:43:03Z

/ok-to-test

dixudx · 2018-03-12T08:21:02Z

cmd/kubeadm/app/discovery/token/token.go

 	for _, endpoint := range endpoints {
 		wg.Add(1)
 		go func(apiEndpoint string) {
+			retries := constants.DiscoveryTimeout / constants.DiscoveryRetryInterval
 			defer wg.Done()
 			wait.Until(func() {
 				fmt.Printf("[discovery] Trying to connect to API Server %q\n", apiEndpoint)


Show the retry times here? So that users can get to know the progress.

dixudx · 2018-03-12T08:22:33Z

cmd/kubeadm/app/discovery/token/token.go

 			defer wg.Done()
 			wait.Until(func() {
 				fmt.Printf("[discovery] Trying to connect to API Server %q\n", apiEndpoint)
 				cfg, err := fetchKubeConfigFunc(apiEndpoint)
 				if err != nil {
 					fmt.Printf("[discovery] Failed to connect to API Server %q: %v\n", apiEndpoint, err)
+					retries--
+					if retries == 0 {
+						resultingError = err


Better to add a customized message to the error, like [discovery] Aborted after retrying xx times: <error msg here>. WDYT?

dixudx · 2018-03-12T08:24:58Z

cmd/kubeadm/app/discovery/token/token_test.go

 			timeout, _ := strconv.Atoi(endpoint)
 			time.Sleep(time.Second * time.Duration(timeout))
 			return kubeconfigutil.CreateBasic(endpoint, "foo", "foo", []byte{}), nil
 		})
+		if err != nil {
+			t.Errorf("failed TestRunForEndpointsAndReturnFirst: unexpected error %v", err)


Just t.Errorf("unexpected error: %v for test %s", err, idx)?

The format follows the message from line 82 of the same file. I can simplify it though (as long as this does not break some test result analysis script somewhere).

Go ahead. This won't break the test.

rosti · 2018-03-12T12:43:55Z

Updated with the suggestions made by @dixudx

dixudx · 2018-03-13T03:24:28Z

cmd/kubeadm/app/discovery/token/token.go

+					if retryCount > retries {
+						resultingError = err
+						close(stopChan)
+						fmt.Printf("[discovery] Abort connecting to API Server %q after %d retries: %v\n", apiEndpoint, retries, err)


How about resultingError = fmt.Errorf("abort connecting to API Server %q after %d retries: %v\n", apiEndpoint, retries, err)?

dixudx · 2018-03-13T03:27:57Z

cmd/kubeadm/app/discovery/token/token_test.go

 			timeout, _ := strconv.Atoi(endpoint)
 			time.Sleep(time.Second * time.Duration(timeout))
 			return kubeconfigutil.CreateBasic(endpoint, "foo", "foo", []byte{}), nil
 		})
+		if err != nil {
+			t.Errorf("unexpected error: %v for TestRunForEndpointsAndReturnFirst", err)


How about t.Errorf("unexpected error: %v for test %s", err, idx)?

We should clearly get to know which test case fails.

rosti · 2018-03-13T10:02:13Z

Again, updated with latest suggestions.

dixudx · 2018-03-14T03:37:13Z

cmd/kubeadm/app/discovery/token/token.go

@@ -206,7 +218,7 @@ func runForEndpointsAndReturnFirst(endpoints []string, fetchKubeConfigFunc func(
 		}(endpoint)
 	}
 	wg.Wait()
-	return resultingKubeConfig
+	return resultingKubeConfig, resultingError


resultingError can be mutated if we've got multiple endpoints, right?

If so, I'd prefer not to return an extra error, which will cause potential issues.
For example, if one of three endpoints is accessible, in this case, we should return an available kubeconfig with nil error, right? Instead of current non-nil resultingError.

Then what we really need to do is to make sure kubeconfig is not nil when using it.

Yes, you are quite right there. Somehow I missed that (probably because I test with a single master, hence a single endpoint)

rosti · 2018-03-14T16:05:40Z

Rewrote the main part of the PR as the previous implementation did not deal too well with multiple API servers.

dixudx

/lgtm

dixudx · 2018-03-15T05:28:38Z

ping @luxas @timothysc for another look. Thanks.

rosti · 2018-03-19T13:20:46Z

/retest

rosti · 2018-03-21T16:37:48Z

Updated the PR by introducing the discoveryTimeout config option. Also updated the PR name, text and release note in accordance with the change.

rosti · 2018-03-21T16:59:25Z

Marking this as a WIP due to a failed test, that requires a bit more thinking to get over.

k8s-github-robot · 2018-03-23T08:59:17Z

[MILESTONENOTIFIER] Milestone Removed From Pull Request

@dixudx @luxas @rosti @timothysc

Important: This pull request was missing labels required for the v1.11 milestone for more than 3 days:

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.
priority: Must specify exactly one of priority/critical-urgent, priority/important-longterm or priority/important-soon.

Help

rosti · 2018-03-26T08:52:27Z

/retest

xiangpengzhao · 2018-04-03T08:45:18Z

cmd/kubeadm/app/discovery/token/token.go

+	select {
+	case <-time.After(timeout):
+		close(stopChan)
+		err := fmt.Errorf("Abort connecting to API servers after timeout of %v", timeout)


s/Abort/abort

xiangpengzhao · 2018-04-03T08:50:04Z

cmd/kubeadm/app/discovery/token/token.go

@@ -205,8 +210,21 @@ func runForEndpointsAndReturnFirst(endpoints []string, fetchKubeConfigFunc func(
 			}, constants.DiscoveryRetryInterval, stopChan)
 		}(endpoint)
 	}
-	wg.Wait()
-	return resultingKubeConfig
+	timeout := constants.DefaultDiscoveryTimeout


Set the default value for field DiscoveryTimeout in SetDefaults_NodeConfiguration instead of defining here?

Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>

rosti · 2018-04-03T11:07:20Z

Updated the PR with @xiangpengzhao 's suggestions

timothysc

I could nit on some style things, but lets get this unblocked now that 1.11 is open
/lgtm
/approve

k8s-ci-robot · 2018-04-03T13:53:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dixudx, rosti, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-04-03T18:04:34Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-04-03T18:33:09Z

Automatic merge from submit-queue (batch tested with PRs 60983, 62012, 61892, 62051, 62067). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot · 2018-04-03T19:05:50Z

@rosti: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce	`230a9c6`	link	`/test pull-kubernetes-e2e-gce`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot assigned luxas and timothysc Mar 9, 2018

k8s-ci-robot requested review from kad and timothysc March 9, 2018 15:29

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 9, 2018

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 10, 2018

timothysc added this to the v1.11 milestone Mar 11, 2018

dixudx reviewed Mar 12, 2018

View reviewed changes

rosti force-pushed the join-timeout branch from ec5edd0 to 4bdef59 Compare March 12, 2018 12:42

dixudx reviewed Mar 13, 2018

View reviewed changes

rosti force-pushed the join-timeout branch from 4bdef59 to a62abd7 Compare March 13, 2018 10:00

dixudx suggested changes Mar 14, 2018

View reviewed changes

rosti force-pushed the join-timeout branch from a62abd7 to a6cb38c Compare March 14, 2018 16:01

k8s-ci-robot assigned dixudx Mar 15, 2018

dixudx reviewed Mar 15, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 15, 2018

k8s-github-robot added the milestone/incomplete-labels label Mar 20, 2018

rosti changed the title ~~kubeadm: Timeout after 5 minutes of join failures~~ kubeadm: Controllable timeout for join failures Mar 21, 2018

rosti force-pushed the join-timeout branch from e776798 to 2ed00ff Compare March 21, 2018 16:52

rosti changed the title ~~kubeadm: Controllable timeout for join failures~~ [WIP] kubeadm: Controllable timeout for join failures Mar 21, 2018

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 21, 2018

rosti force-pushed the join-timeout branch from 2ed00ff to fb7a8a9 Compare March 22, 2018 13:50

rosti changed the title ~~[WIP] kubeadm: Controllable timeout for join failures~~ kubeadm: Controllable timeout for join failures Mar 22, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 22, 2018

k8s-github-robot added milestone/removed and removed milestone/incomplete-labels labels Mar 23, 2018

k8s-github-robot removed this from the v1.11 milestone Mar 23, 2018

rosti force-pushed the join-timeout branch from fb7a8a9 to 9b29aea Compare March 23, 2018 12:24

rosti force-pushed the join-timeout branch from 9b29aea to 36cb1fd Compare March 30, 2018 08:11

xiangpengzhao reviewed Apr 3, 2018

View reviewed changes

rosti force-pushed the join-timeout branch from 36cb1fd to 454f793 Compare April 3, 2018 11:02

kubeadm: Introduce controllable timeout on join

230a9c6

Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>

rosti force-pushed the join-timeout branch from 454f793 to 230a9c6 Compare April 3, 2018 11:05

timothysc approved these changes Apr 3, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 3, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 3, 2018

k8s-github-robot merged commit 7973c54 into kubernetes:master Apr 3, 2018

rosti deleted the join-timeout branch April 26, 2018 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: Controllable timeout for join failures #60983

kubeadm: Controllable timeout for join failures #60983

rosti commented Mar 9, 2018 •

edited

Loading

k8s-ci-robot commented Mar 9, 2018

BenTheElder commented Mar 10, 2018

dixudx Mar 12, 2018

dixudx Mar 12, 2018

dixudx Mar 12, 2018

rosti Mar 12, 2018

dixudx Mar 12, 2018

rosti commented Mar 12, 2018

dixudx Mar 13, 2018

dixudx Mar 13, 2018

rosti commented Mar 13, 2018

dixudx Mar 14, 2018 •

edited

Loading

rosti Mar 14, 2018

rosti commented Mar 14, 2018

dixudx left a comment

dixudx commented Mar 15, 2018

rosti commented Mar 19, 2018

rosti commented Mar 21, 2018

rosti commented Mar 21, 2018

k8s-github-robot commented Mar 23, 2018

rosti commented Mar 26, 2018

xiangpengzhao Apr 3, 2018

xiangpengzhao Apr 3, 2018

rosti commented Apr 3, 2018

timothysc left a comment

k8s-ci-robot commented Apr 3, 2018

k8s-github-robot commented Apr 3, 2018

k8s-github-robot commented Apr 3, 2018

k8s-ci-robot commented Apr 3, 2018 •

edited

Loading

kubeadm: Controllable timeout for join failures #60983

kubeadm: Controllable timeout for join failures #60983

Conversation

rosti commented Mar 9, 2018 • edited Loading

k8s-ci-robot commented Mar 9, 2018

BenTheElder commented Mar 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Mar 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Mar 13, 2018

dixudx Mar 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Mar 14, 2018

dixudx left a comment

Choose a reason for hiding this comment

dixudx commented Mar 15, 2018

rosti commented Mar 19, 2018

rosti commented Mar 21, 2018

rosti commented Mar 21, 2018

k8s-github-robot commented Mar 23, 2018

rosti commented Mar 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti commented Apr 3, 2018

timothysc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 3, 2018

k8s-github-robot commented Apr 3, 2018

k8s-github-robot commented Apr 3, 2018

k8s-ci-robot commented Apr 3, 2018 • edited Loading

rosti commented Mar 9, 2018 •

edited

Loading

dixudx Mar 14, 2018 •

edited

Loading

k8s-ci-robot commented Apr 3, 2018 •

edited

Loading