Fix nil pointer error in nodevolumelimits csi logging #115179

sunnylovestiramisu · 2023-01-18T20:15:27Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fix nil pointer error in logging: #115178

Which issue(s) this PR fixes:

Fixes #115178

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix 1.25 regression causing a nil pointer error in nodevolumelimits csi logging

k8s-ci-robot · 2023-01-18T20:15:36Z

@sunnylovestiramisu: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2023-01-18T20:26:15Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi.go

@@ -231,8 +231,12 @@ func (pl *CSILimits) checkAttachableInlineVolume(vol v1.Volume, csiNode *storage
 		return fmt.Errorf("looking up provisioner name for volume %v: %w", vol, err)
 	}
 	if !isCSIMigrationOn(csiNode, inTreeProvisionerName) {
+		csiNodeName := ""


Can a unit test with nil CSINode be added?

There is no existing test set up but we can add a new TestCheckAttachableInlineVolume

I think the existing test may need some tweaks to be able to pass in a nil CSINode:
https://github.com/kubernetes/kubernetes/blob/818dfc6f4147d36522dd4bf18146efec4832f1c1/pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go#L483

Modified the test and now if it is a nil node the unit test throws the node not found.

Actually I needed to modify the getFakeCSINodeLister to fake returning a nil csi node.

I guess this change is not necessary if we return the error?

sunnylovestiramisu · 2023-01-18T23:34:28Z

/assign @msau42

msau42 · 2023-01-18T23:38:38Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go

@@ -661,7 +674,10 @@ func getNodeWithPodAndVolumeLimits(limitSource string, pods []*v1.Pod, limit int
 	default:
 		// Do nothing.
 	}
-
-	nodeInfo.SetNode(node)
+	if isNilNode {


We want csiNode to be nil, not node

Then we should modify the getFakeCSINodeLister so that when we call Get the csiNode is nil.

msau42 · 2023-01-19T03:34:42Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go

@@ -598,8 +610,8 @@ func getFakeCSIStorageClassLister(scName, provisionerName string) fakeframework.
 	}
 }

-func getFakeCSINodeLister(csiNode *storagev1.CSINode) fakeframework.CSINodeLister {
-	if csiNode != nil {
+func getFakeCSINodeLister(csiNode *storagev1.CSINode, isNilCsiNode bool) fakeframework.CSINodeLister {


It looks like the function already handles a nil csiNode input. Do you need to add this extra field?

Because we cannot set the csiNode to nil before this function, there is a set up step for csiNode.spec.Something. The setup will fail?

actually I think limitSource = "node" should already cause csiNode to be nil. If you run the existing tests with -v 5, do you encounter the error?

I just tried to modify the test case for limitSource as node in the master branch:

{ newPod: inTreeOneVolPod, existingPods: []*v1.Pod{inTreeTwoVolPod}, filterName: "csi", maxVols: 2, driverNames: []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, limitSource: "node", test: "should count in-tree volumes if migration is enabled", }

The test just passed without panic.

The test case should use a pod specification using an inline volume.

Also it looks like the FakeLister doesn't return nil in a "not found" scenario: https://github.com/kubernetes/kubernetes/blob/6ec579904c6120b961ed0a1752cd9f92dfb5124c/pkg/scheduler/framework/fake/listers.go#L264

inTreeInlineVolPod on master branch testing also did not trigger the painc. Made the change for the fake lister, some other existing tests start to fail:

--- FAIL: TestCSILimits/should_count_in-tree_volumes_if_migration_is_enabled (0.00s) --- FAIL: TestCSILimits/should_count_unbound_in-tree_volumes_if_migration_is_enabled (0.00s) --- FAIL: TestCSILimits/should_count_in-tree_inline_volumes_if_migration_is_enabled (0.00s) --- FAIL: TestCSILimits/should_count_in-tree_and_csi_volumes_if_migration_is_enabled_(when_scheduling_in-tree_volumes) (0.00s) --- FAIL: TestCSILimits/should_count_in-tree,_inline_and_csi_volumes_if_migration_is_enabled_(when_scheduling_in-tree_volumes) (0.00s) --- FAIL: TestCSILimits/should_count_in-tree_and_csi_volumes_if_migration_is_enabled_(when_scheduling_csi_volumes) (0.00s)

alculquicondor · 2023-01-19T14:12:45Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi.go

@@ -97,6 +97,7 @@ func (pl *CSILimits) Filter(ctx context.Context, _ *framework.CycleState, pod *v
 	if err != nil {
 		// TODO: return the error once CSINode is created by default (2 releases)
 		klog.V(5).InfoS("Could not get a CSINode object for the node", "node", klog.KObj(node), "err", err)
+		return framework.AsStatus(err)


This will cause the scheduler/cluster-autoscaler to retry immediately, as opposed to returning an UnschedulableAndUnresolvable status.

Is retrying the expected behavior?

I think the comment is incorrect as discussed in #107787 (comment).

Autoscaler doesn't simulate CSINode objects today, so we can't return an error otherwise autoscaler will fail forever.

Oh wait, sorry, this is my miss, I returned the error here to checkout the logs I added in the test. But after I verified the code path I only removed the logs but forgot to remove this line.

alculquicondor · 2023-01-19T14:13:38Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi.go

@@ -231,8 +231,12 @@ func (pl *CSILimits) checkAttachableInlineVolume(vol v1.Volume, csiNode *storage
 		return fmt.Errorf("looking up provisioner name for volume %v: %w", vol, err)
 	}
 	if !isCSIMigrationOn(csiNode, inTreeProvisionerName) {
+		csiNodeName := ""


I guess this change is not necessary if we return the error?

alculquicondor · 2023-01-19T17:06:08Z

pkg/scheduler/framework/plugins/nodevolumelimits/non_csi.go

+	// ErrReasonNodeNotFound is used for node not found error.
+	ErrReasonNodeNotFound = "node not found"


sunnylovestiramisu · 2023-01-24T18:09:09Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi.go

@@ -231,8 +232,12 @@ func (pl *CSILimits) checkAttachableInlineVolume(vol v1.Volume, csiNode *storage
 		return fmt.Errorf("looking up provisioner name for volume %v: %w", vol, err)
 	}
 	if !isCSIMigrationOn(csiNode, inTreeProvisionerName) {
+		csiNodeName := ""
+		if csiNode != nil {


The mock is not working as expected after changing to a list, even the test case said that migration is enabled, on line old 291, new 296:

if !isCSIMigrationOn(csiNode, pluginName) { klog.V(5).InfoS("CSI Migration of plugin is not enabled", "plugin", pluginName) return "", "" }

This got returned because csiNode is nil.

sunnylovestiramisu · 2023-01-24T21:04:17Z

It turned out the original test setup did not reflect the reality, the name of csi-node and node should be the same.

Added extra logs locally( please see here ) to test the csi node == nil:

// go test -v -run TestCSILimits
=== RUN   TestCSILimits/nil_csi_node
===== haha I am a nil csi node  =====

sunnylovestiramisu · 2023-01-24T21:18:30Z

/test pull-kubernetes-unit
/meow

k8s-ci-robot · 2023-01-24T21:18:32Z

@sunnylovestiramisu:

In response to this:

/test pull-kubernetes-unit
/meow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2023-01-25T02:01:31Z

pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go

+			driverNames:      []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName},
+			migrationEnabled: true,
+			isNilCsiNode:     true,
+			limitSource:      "csinode",


are you able to repro the error if you set limitSource to node and without needing isNilCsiNode?

We need the isNilCsiNode to generate the nil pointer exception? I commented out the csiNodeName and log the csiNode.Name the test returns error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x13ba74d]

when limitSource == node, I believe that should also cause csiNode to be nil: https://github.com/kubernetes/kubernetes/blob/250497ebca681fa8c054cac45bb5d8438b67d7a4/pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go#L663

I tried again the same test case as the last week's code review comment above:

{ newPod: inTreeOneVolPod, existingPods: []*v1.Pod{inTreeTwoVolPod}, filterName: "csi", maxVols: 2, driverNames: []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, limitSource: "node", test: "should count in-tree volumes if migration is enabled", wantStatus: framework.NewStatus(framework.Unschedulable, ErrReasonMaxVolumeCountExceeded), }

It did not trigger the csiNode == nil, it did not go into the if !isCSIMigrationOn(csiNode, inTreeProvisionerName) code path. I do not think the changes from the master branch fixed the test setup? So the change of limitSource still does not change the test result. :(

I checked out your PR, reverted csi.go, and tweaked the test inputs and was able to repro the crash:

--- a/pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go +++ b/pkg/scheduler/framework/plugins/nodevolumelimits/csi_test.go @@ -313,8 +313,8 @@ func TestCSILimits(t *testing.T) { maxVols: 2, driverNames: []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, - isNilCsiNode: true, - limitSource: "csinode", + isNilCsiNode: false, + limitSource: "node", test: "nil csi node", },

Does this mean we do not need the isNilCsiNode params and just fix the csi-node-name to node-name should fix the test?

We need the isNilCsiNode to generate the nil pointer exception? I commented out the csiNodeName and log the csiNode.Name the test returns error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x13ba74d]

The above was ran with:

{ newPod: inTreeInlineVolPod, existingPods: []*v1.Pod{inTreeTwoVolPod}, filterName: "csi", maxVols: 2, driverNames: []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, isNilCsiNode: true, limitSource: "csinode", }

without the change:

csiNodeName := "" if csiNode != nil { csiNodeName = csiNode.Name }

also generated the nil pointer exception, do we need another test case specific to

isNilCsiNode: false, limitSource: "node",

sunnylovestiramisu · 2023-01-25T22:19:05Z

Tests failed with bytes-like object, not str, which looks like a python2 vs python3 issue, a few other PRs also running into the same error:
kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1116

msau42 · 2023-01-26T00:58:15Z

/lgtm
/assign @alculquicondor

k8s-ci-robot · 2023-01-26T00:58:21Z

LGTM label has been added.

Git tree hash: 2a1432bde8ab29d6e83b1a0bde51450313ebfff1

msau42 · 2023-01-26T01:00:16Z

Can you add a release note for the fix? We'll want to cherry pick this.

alculquicondor

LGTM from scheduling

msau42 · 2023-01-26T18:55:34Z

/approve

alculquicondor · 2023-01-26T18:55:42Z

/approve

k8s-ci-robot · 2023-01-26T18:55:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, msau42, sunnylovestiramisu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/scheduler/OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ick-of-#115179-upstream-release-1.25 Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging

…ick-of-#115179-upstream-release-1.26 Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging

openshift-merge-robot · 2023-11-22T01:24:13Z

Fix included in accepted release 4.13.0-0.nightly-2023-11-21-212406

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 18, 2023

k8s-ci-robot requested review from denkensk and sanposhiho January 18, 2023 20:17

msau42 reviewed Jan 18, 2023

View reviewed changes

sunnylovestiramisu force-pushed the fixPanic branch from 818dfc6 to ce3ec84 Compare January 18, 2023 22:46

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 18, 2023

sunnylovestiramisu force-pushed the fixPanic branch 5 times, most recently from 5e730ac to 477a4ac Compare January 18, 2023 23:13

k8s-ci-robot assigned msau42 Jan 18, 2023

msau42 reviewed Jan 18, 2023

View reviewed changes

sunnylovestiramisu force-pushed the fixPanic branch 2 times, most recently from fe28ec2 to bbd38aa Compare January 19, 2023 00:46

msau42 reviewed Jan 19, 2023

View reviewed changes

alculquicondor reviewed Jan 19, 2023

View reviewed changes

sunnylovestiramisu force-pushed the fixPanic branch from bbd38aa to 8576478 Compare January 19, 2023 17:03

alculquicondor reviewed Jan 19, 2023

View reviewed changes

sunnylovestiramisu commented Jan 24, 2023

View reviewed changes

sunnylovestiramisu force-pushed the fixPanic branch from 622be72 to 250497e Compare January 24, 2023 21:00

msau42 reviewed Jan 25, 2023

View reviewed changes

Fix nil pointer error in nodevolumelimits csi logging

5e2f12e

sunnylovestiramisu force-pushed the fixPanic branch from 250497e to 5e2f12e Compare January 25, 2023 17:38

k8s-ci-robot assigned alculquicondor Jan 26, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 26, 2023

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jan 26, 2023

alculquicondor reviewed Jan 26, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 26, 2023

k8s-ci-robot merged commit 97ab147 into kubernetes:master Jan 26, 2023

k8s-ci-robot added this to the v1.27 milestone Jan 26, 2023

sunnylovestiramisu deleted the fixPanic branch January 26, 2023 21:28

This was referenced Jan 26, 2023

Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging #115347

Merged

Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging #115348

Merged

k8s-ci-robot added a commit that referenced this pull request Feb 10, 2023

Merge pull request #115348 from sunnylovestiramisu/automated-cherry-p…

30607e7

…ick-of-#115179-upstream-release-1.25 Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging

k8s-ci-robot added a commit that referenced this pull request Feb 10, 2023

Merge pull request #115347 from sunnylovestiramisu/automated-cherry-p…

e115dfe

…ick-of-#115179-upstream-release-1.26 Automated cherry pick of #115179: Fix nil pointer error in nodevolumelimits csi logging

tbak mentioned this pull request Feb 17, 2023

Seg Fault in nodeVolumeLimits of scheduler plugin causing panic #115178

Closed

		// ErrReasonNodeNotFound is used for node not found error.
		ErrReasonNodeNotFound = "node not found"

Fix nil pointer error in nodevolumelimits csi logging #115179

Fix nil pointer error in nodevolumelimits csi logging #115179

Conversation

sunnylovestiramisu commented Jan 18, 2023 • edited by liggitt

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Jan 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 Jan 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu commented Jan 18, 2023

Choose a reason for hiding this comment

sunnylovestiramisu Jan 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Jan 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Jan 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Jan 24, 2023 • edited

Choose a reason for hiding this comment

sunnylovestiramisu commented Jan 24, 2023

sunnylovestiramisu commented Jan 24, 2023

k8s-ci-robot commented Jan 24, 2023

Choose a reason for hiding this comment

sunnylovestiramisu Jan 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Jan 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Jan 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu commented Jan 25, 2023

msau42 commented Jan 26, 2023

k8s-ci-robot commented Jan 26, 2023

msau42 commented Jan 26, 2023

alculquicondor left a comment

Choose a reason for hiding this comment

msau42 commented Jan 26, 2023

alculquicondor commented Jan 26, 2023

k8s-ci-robot commented Jan 26, 2023

openshift-merge-robot commented Nov 22, 2023

sunnylovestiramisu commented Jan 18, 2023 •

edited by liggitt

msau42 Jan 18, 2023 •

edited

sunnylovestiramisu Jan 18, 2023 •

edited

sunnylovestiramisu Jan 19, 2023 •

edited

sunnylovestiramisu Jan 19, 2023 •

edited

sunnylovestiramisu Jan 24, 2023 •

edited

sunnylovestiramisu Jan 25, 2023 •

edited

sunnylovestiramisu Jan 25, 2023 •

edited

sunnylovestiramisu Jan 25, 2023 •

edited