WIP Fixing leaking gRPC sessions between CSI driver and CSI components #64380

sbezverk · 2018-05-27T21:11:41Z

k8s-ci-robot · 2018-05-27T21:11:42Z

@sbezverk: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2018-05-27T21:12:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sbezverk
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: saad-ali

Assign the PR to them by writing /assign @saad-ali in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

pkg/volume/csi/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbezverk · 2018-05-27T23:42:32Z

@saad-ali @vladimirvivien I had to remove unit tests at least for now, because with the way they were built, I could not find how to expose grpc connection without breaking them, and also these unit tests look more like e2e tests. Since we have already a stable e2e test, I hope you would agree if I refactor (possibly in a separate PR) unit tests to test functions and not e2e flow.

k8s-ci-robot · 2018-05-28T00:27:33Z

@sbezverk: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce-device-plugin-gpu	`3f892e2`	link	`/test pull-kubernetes-e2e-gce-device-plugin-gpu`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sbezverk · 2018-05-28T02:53:58Z

/test pull-kubernetes-e2e-gce

vladimirvivien

The replacement code looks good. I added some suggestions. However, concerned that lot of test code seemed to be deleted.

vladimirvivien · 2018-05-28T14:48:25Z

pkg/volume/csi/csi_client.go

-		c.idClient = csipb.NewIdentityClient(conn)
-		c.nodeClient = csipb.NewNodeClient(conn)
-		c.ctrlClient = csipb.NewControllerClient(conn)
+func (c *csiDriverClient) IdentityRequest() (*grpc.ClientConn, csipb.IdentityClient, error) {


@sbezverk I would call method IdentidyClient, instead of IdentityRequest, since it's used to create the client.

How about NewIdentityClient()?

vladimirvivien · 2018-05-28T14:50:03Z

pkg/volume/csi/csi_client.go

+	return conn, c.idClient, nil
+}
+
+func (c *csiDriverClient) NodeRequest() (*grpc.ClientConn, csipb.NodeClient, error) {


Call method NodeClient, instead of NodeRequest, since it's used to create the client.

How about NewNodeClient()?

vladimirvivien · 2018-05-28T14:50:33Z

pkg/volume/csi/csi_client.go


-		// set supported version
+func (c *csiDriverClient) ControllerRequest() (*grpc.ClientConn, csipb.ControllerClient, error) {


I would call method ControllerClient, instead of ControllerRequest, since it's used to create the client.

How about NewControllerClient()?

vladimirvivien · 2018-05-28T14:55:06Z

pkg/volume/csi/csi_client_test.go

@@ -1,173 +0,0 @@
-/*


Would be a good idea to have some tests for the newly added client code.

Will add some tests.

vladimirvivien · 2018-05-28T14:55:55Z

pkg/volume/csi/csi_mounter_test.go

-	testPodUID = types.UID("test-pod")
-)
-
-func TestMounterGetPath(t *testing.T) {


Why are the mounter tests deleted ?

vladimirvivien · 2018-05-28T14:56:20Z

pkg/volume/csi/csi_plugin_test.go

-)
-
-// create a plugin mgr to load plugins and setup a fake client
-func newTestPlugin(t *testing.T) (*csiPlugin, string) {


Why are plugin tests deleted ?

all CSI tests are using setupClient func from csi_client_tests which is using fake package to mock the driver. So far I could not figure out a way to make it working with new Identity/Node/Controller clients.
I always get:

panic: interface conversion: csi.NodeClient is *csi.nodeClient, not *fake.NodeClient [recovered] panic: interface conversion: csi.NodeClient is *csi.nodeClient, not *fake.NodeClient

Looks like the clients are now being passed around by reference, so the test code should be updated to match that.

sbezverk · 2018-05-30T14:39:47Z

The issue is being worked on in a different pr, closing this one.

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. CSI fix for gRPC conn leak **What this PR does / why we need it**: This PR is a bug fix for leaky gRPC connection that never closes (see issue #64341 for detail) **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #64341 This fix was originally started with PR #64380 ```release-note NONE ```

Initial code load

fae5774

k8s-ci-robot requested review from msau42 and verult May 27, 2018 21:12

sbezverk added 2 commits May 27, 2018 17:17

Fixing gofmt

c5d094a

Remove conflicting unit tests

218e131

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 27, 2018

Fixing bazel failure

3f892e2

vladimirvivien reviewed May 28, 2018

View reviewed changes

neolit123 mentioned this pull request May 29, 2018

pull-kubernetes-e2e-gce-device-plugin-gpu test always fail #64384

Closed

sbezverk closed this May 30, 2018

vladimirvivien mentioned this pull request May 30, 2018

CSI fix for gRPC conn leak #64519

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Fixing leaking gRPC sessions between CSI driver and CSI components #64380

WIP Fixing leaking gRPC sessions between CSI driver and CSI components #64380

sbezverk commented May 27, 2018

k8s-ci-robot commented May 27, 2018

k8s-ci-robot commented May 27, 2018

sbezverk commented May 27, 2018

k8s-ci-robot commented May 28, 2018 •

edited

sbezverk commented May 28, 2018

vladimirvivien left a comment

vladimirvivien May 28, 2018

sbezverk May 28, 2018

saad-ali May 30, 2018

vladimirvivien May 28, 2018

sbezverk May 28, 2018

saad-ali May 30, 2018

vladimirvivien May 28, 2018

sbezverk May 28, 2018

saad-ali May 30, 2018

vladimirvivien May 28, 2018

sbezverk May 28, 2018

vladimirvivien May 28, 2018

vladimirvivien May 28, 2018

sbezverk May 28, 2018

saad-ali May 30, 2018

sbezverk commented May 30, 2018


		// set supported version
		func (c csiDriverClient) ControllerRequest() (grpc.ClientConn, csipb.ControllerClient, error) {

WIP Fixing leaking gRPC sessions between CSI driver and CSI components #64380

WIP Fixing leaking gRPC sessions between CSI driver and CSI components #64380

Conversation

sbezverk commented May 27, 2018

k8s-ci-robot commented May 27, 2018

k8s-ci-robot commented May 27, 2018

sbezverk commented May 27, 2018

k8s-ci-robot commented May 28, 2018 • edited

sbezverk commented May 28, 2018

vladimirvivien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbezverk commented May 30, 2018

k8s-ci-robot commented May 28, 2018 •

edited