Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #89589

jsafrane · 2020-03-27T17:53:00Z

This is fixed version of #88000 when @davidz627 is out.

fixed unit test speed
added logs
probe API server by getting a dummy CSINode object. Kubelet may have limited permission during TLS bootstrap and we need kubelet with real permission to publish CSINode.

I kept David's commit there, so he gets credit for the fix.

If APIServer takes a long time to initialize (faster than the timeout for CSI Node intialization) the Kubelet may kill itself before the APIServer has time to come up. This PR makes it so that the goroutine that initializes CSINode will poll and wait for APIServer initialization before starting the exponential backoff to create CSINode objects.

Fixes: #87964

/cc @misterikkit @msau42 @rphillips @marun @tedyu
/kind bug

CSINode initialization does not crash kubelet on startup when APIServer is not reachable or kubelet has not the right credentials yet.

…Kubelet init

To speed up unit tests and add more observability when things go wrong.

k8s-ci-robot · 2020-03-27T17:55:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/volume/csi/OWNERS~~ [jsafrane]
~~pkg/volume/testing/OWNERS~~ [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jsafrane · 2020-03-27T17:59:02Z

/priority important-soon

jsafrane · 2020-03-27T18:17:20Z

cc @liggitt to check CSINode probe. Is here a better way how to check TLS bootstrap is finished and kubelet is ready to start for real?

tedyu · 2020-03-27T18:20:53Z

pkg/volume/csi/csi_plugin.go

@@ -914,3 +919,30 @@ func highestSupportedVersion(versions []string) (*utilversion.Version, error) {
 	}
 	return highestSupportedVersion, nil
 }
+
+// waitForAPIServerForever waits forever to get the APIServer Version as a proxy


Please modify this comment to be consistent with the node retrieval on line 934

tedyu · 2020-03-27T18:22:00Z

pkg/volume/csi/csi_plugin.go

+		// https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/
+		// Using a dummy node name, IsNotFound error is expected. IsForbidden or timeout are bad.
+		_, lastErr = client.StorageV1().CSINodes().Get(context.TODO(), nonExistingNodeName, meta.GetOptions{})
+		if lastErr == nil || apierrors.IsNotFound(lastErr) {


lastErr shouldn't be nil if the node is non-existent :-)

you never know :-)

One could ~~crash~~ break whole cluster by creating a node with nonExistingNodeName.

Since the node may exist, we can give it name such as possiblyExistingNodeName

jsafrane · 2020-03-28T08:17:57Z

Trick with getting CSINode does not work:

Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "nonExistingNodeName" is forbidden: User "system:node:e2e-29112e7941-674b9-minion-group-1djj" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope: can only access CSINode with the same name as the requesting node

tedyu · 2020-03-28T18:23:42Z

With the following change to csi_plugin (nonExistingNodeName is no longer used), test suites pass:

		err := waitForAPIServerForever(kubeClient, host.GetHostName())
...
func waitForAPIServerForever(client clientset.Interface, host string) error {
...
		_, lastErr = client.StorageV1().CSINodes().Get(context.TODO(), host, meta.GetOptions{})

jsafrane · 2020-03-30T08:12:49Z

Updated to get CSINode of the current node.

k8s-ci-robot · 2020-03-30T08:43:31Z

@jsafrane: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kind-ipv6	`8bdbd4d`	link	`/test pull-kubernetes-e2e-kind-ipv6`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

tedyu · 2020-03-30T12:24:43Z

/lgtm

jsafrane · 2020-04-03T07:45:57Z

Edited the release note a bit to emphasize kubelet panic.

…89-upstream-release-1.17 Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode

…89-upstream-release-1.18 Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode

davidz627 and others added 2 commits March 27, 2020 18:27

Wait for APIServer 'ok' forever during CSINode initialization during …

8205f81

…Kubelet init

Fix CSINodeInfo startup

8bdbd4d

To speed up unit tests and add more observability when things go wrong.

k8s-ci-robot requested review from marun, misterikkit, msau42, rphillips and tedyu March 27, 2020 17:53

jsafrane mentioned this pull request Mar 27, 2020

Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #88000

Closed

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 27, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 27, 2020

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 27, 2020

jsafrane force-pushed the fix-node-startup branch 2 times, most recently from 224dd86 to 60e901a Compare March 27, 2020 18:15

tedyu reviewed Mar 27, 2020

View reviewed changes

jsafrane force-pushed the fix-node-startup branch from 60e901a to eacb857 Compare March 27, 2020 18:30

jsafrane force-pushed the fix-node-startup branch from eacb857 to 8bdbd4d Compare March 30, 2020 08:13

k8s-ci-robot assigned tedyu Mar 30, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2020

k8s-ci-robot merged commit e178cac into kubernetes:master Mar 30, 2020

k8s-ci-robot added this to the v1.19 milestone Mar 30, 2020

This was referenced Mar 30, 2020

Bug 1817382: UPSTREAM: 89589: Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init openshift/origin#24798

Merged

Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode #89809

Merged

jsafrane mentioned this pull request Apr 3, 2020

Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode #89810

Merged

k8s-ci-robot added a commit that referenced this pull request Apr 17, 2020

Merge pull request #89810 from jsafrane/automated-cherry-pick-of-#895…

bd4cc79

…89-upstream-release-1.17 Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode

k8s-ci-robot added a commit that referenced this pull request Apr 17, 2020

Merge pull request #89809 from jsafrane/automated-cherry-pick-of-#895…

e97c570

…89-upstream-release-1.18 Automated cherry pick of #89589: Wait for APIServer 'ok' forever during CSINode

hakman mentioned this pull request Jun 10, 2020

Incorrect disk size and usage detection blocks e2e from starting with Kops #91982

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #89589

Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #89589

jsafrane commented Mar 27, 2020 •

edited

k8s-ci-robot commented Mar 27, 2020

jsafrane commented Mar 27, 2020

jsafrane commented Mar 27, 2020

tedyu Mar 27, 2020

jsafrane Mar 27, 2020

tedyu Mar 27, 2020

jsafrane Mar 27, 2020

jsafrane Mar 27, 2020 •

edited

tedyu Mar 27, 2020

jsafrane commented Mar 28, 2020

tedyu commented Mar 28, 2020 •

edited

jsafrane commented Mar 30, 2020

k8s-ci-robot commented Mar 30, 2020 •

edited

tedyu commented Mar 30, 2020

jsafrane commented Apr 3, 2020

Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #89589

Wait for APIServer 'ok' forever during CSINode initialization during Kubelet init #89589

Conversation

jsafrane commented Mar 27, 2020 • edited

k8s-ci-robot commented Mar 27, 2020

jsafrane commented Mar 27, 2020

jsafrane commented Mar 27, 2020

tedyu Mar 27, 2020

Choose a reason for hiding this comment

jsafrane Mar 27, 2020

Choose a reason for hiding this comment

tedyu Mar 27, 2020

Choose a reason for hiding this comment

jsafrane Mar 27, 2020

Choose a reason for hiding this comment

jsafrane Mar 27, 2020 • edited

Choose a reason for hiding this comment

tedyu Mar 27, 2020

Choose a reason for hiding this comment

jsafrane commented Mar 28, 2020

tedyu commented Mar 28, 2020 • edited

jsafrane commented Mar 30, 2020

k8s-ci-robot commented Mar 30, 2020 • edited

tedyu commented Mar 30, 2020

jsafrane commented Apr 3, 2020

jsafrane commented Mar 27, 2020 •

edited

jsafrane Mar 27, 2020 •

edited

tedyu commented Mar 28, 2020 •

edited

k8s-ci-robot commented Mar 30, 2020 •

edited