Fix 'Schedulercache is corrupted' error

[liggitt]

Fixes #50916

If an Assume()ed pod is Add()ed with a different nodeName, the podStates view of the pod is not corrected to reflect the actual nodeName. On the next Update(), the scheduler observes the mismatch and process exits.

Fixed 'Schedulercache is corrupted' error in kube-scheduler

[liggitt]

cc @kubernetes/sig-scheduling-bugs @kubernetes/sig-scheduling-pr-reviews

[liggitt]

/retest

[timothysc]

/assign @wojtek-t

[timothysc]

/lgtm - minor comment.

[timothysc]

Why do you up the ttl?

[liggitt]

this is simulating passing of time to ensure the added pod doesn't expire. was copy/paste from previous testcase, though, and not really relevant to this one, so I can remove it

[k8s-cherrypick-bot]

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

[liggitt]

updated to remove unrelated ttl bit from unit test

[cblecker]

CRD Flake
/test pull-kubernetes-unit

[bsalamat]

Your change looks good to me, I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

[bsalamat]

s/AddPod/UpdatePod/

[liggitt]

fixed gofmt error and test message, re-tagging

[liggitt]

I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

It was just the podStates' version of the pod that wasn't getting updated. The removePod(currState.pod)/addPod(pod) dance corrects the per-node accounting already (and the unit test I added demonstrates the per-node accounting is corrected):

kubernetes/plugin/pkg/scheduler/schedulercache/cache.go

Lines 234 to 241 in 454074d

	case ok && cache.assumedPods[key]:
	if currState.pod.Spec.NodeName != pod.Spec.NodeName {
	// The pod was added to a different node than it was assumed to.
	glog.Warningf("Pod %v assumed to a different node than added to.", key)
	// Clean this up.
	cache.removePod(currState.pod)
	cache.addPod(pod)
	}

[cblecker]

@liggitt looks like there's still a gofmt issue on ./plugin/pkg/scheduler/schedulercache/cache_test.go

[bsalamat]

Thanks, @liggitt!

/lgtm

[liggitt]

actually fixed gofmt error

[cblecker]

reapplying LGTM
/lgtm

[k8s-merge-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cblecker, liggitt, timothysc

Associated issue: 50916

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• plugin/pkg/scheduler/OWNERS [bsalamat,timothysc]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[cblecker]

uhhh..

W1107 23:53:54.719] Run: ('bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh')
W1107 23:53:54.722] bash: line 0: cd: kubernetes: No such file or directory

That's an odd unit test failure. Let's retry.
/test pull-kubernetes-unit

[cblecker]

unit test failing: #55276

[cblecker]

fingers crossed
/test pull-kubernetes-unit

[liggitt]

TestCRD flake
/test pull-kubernetes-unit

[k8s-merge-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-merge-robot]

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.