Fix incorrect call to 'bind' in scheduler

[julia-stripe]

I previously submitted #49661 -- I'm not sure if that PR is too big or what, but this is an attempt at a smaller PR that makes progress on the same issue and is easier to review.

What this PR does / why we need it:

In this refactor (ecb962e#diff-67f2b61521299ca8d8687b0933bbfb19R223) the scheduler code was refactored into separate bind and assume functions. When that happened, bind was called with pod as an argument. The argument to bind should be the assumed pod, not the original pod. Evidence that assumedPod is the correct argument bind and not pod:

kubernetes/plugin/pkg/scheduler/scheduler.go

Lines 229 to 234 in 80f26fa

	if err := sched.config.SchedulerCache.FinishBinding(&assumed); err != nil {
	glog.Errorf("scheduler cache FinishBinding failed: %v", err)
	}
	if err != nil {
	glog.V(1).Infof("Failed to bind pod: %v/%v", pod.Namespace, pod.Name)
	if err := sched.config.SchedulerCache.ForgetPod(&assumed); err != nil {

. (and it says assumed in the function signature for bind, even though it's not called with the assumed pod as an argument).

This is an issue (and causes #49314, where pods that fail to bind to a node get stuck indefinitely) in the following scenario:

1. The pod fails to bind to the node
2. bind calls ForgetPod with the pod argument
3. since ForgetPod is expecting the assumed pod as an argument (because that's what's in the scheduler cache), it fails with an error like scheduler cache ForgetPod failed: pod test-677550-rc-edit-namespace/nginx-jvn09 state was assumed on a different node
4. The pod gets lost forever because of some incomplete error handling (which I haven't addressed here in the interest of making a simpler PR)

In this PR I've fixed the call to bind and modified the tests to make sure that ForgetPod gets called with the correct argument (the assumed pod) when binding fails.

Which issue this PR fixes: fixes #49314

Special notes for your reviewer:

Release note:

Fix bug in scheduler that caused initially unschedulable pods to stuck in Pending state forever.

[k8s-ci-robot]

Hi @julia-stripe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[cblecker]

/ok-to-test

[julia-stripe]

/retest

[dchen1107]

From the original issue description, it is a serial regression for 1.7 release, and we should patch it. I marked this pr for cherrypick-candidate.

@julia-stripe Looks like there are two prs (this one and #49661) addressing the same issue. I didn't see much difference from both prs. Could you please close one?

[dchen1107]

The pr looks good to me based on the problem described, and I prefer someone from @kubernetes/sig-scheduling-pr-reviews to take a look too. @bsalamat

[k82cn]

/assign

[bsalamat]

Isn't this the same as before.

[k82cn]

we used pod directly before the fix; and it was assumedPod before the refactor.

[wojtek-t]

It's not the same - assume() method above is modifying the given pod.

[bsalamat]

I meant this piece of code is logically the same in this PR.

[bsalamat]

Is sched.assume() going to modify assumedPod?

[wojtek-t]

@bsalamat I don't understand this question. Yes it is modifying. Yes it is what we are trying to achieve.

[bsalamat]

Thanks, @wojtek-t! I don't know why I missed it. I guess github UI still confuses me.

[wanghaoran1988]

Why you make a copy here? seems you moved the copy from inside the assume func here.

[k82cn]

we should not use obj from cache directly.

[julia-stripe]

Basically we need to make a copy somewhere and @k82cn suggested it would be more clear if we made the copy outside of the assume function. (which I agree with)

[wanghaoran1988]

/cc @aveshagarwal

[davidopp]

@wojtek-t

[wojtek-t]

Thanks a lot for this PR. I added some minor nits but this overall looks good to me.

[k82cn]

@wojtek-t , can you also help to review #49661 ? I think FinishBinding & ForgetPod is also important to 'cherrypick'.

[wojtek-t]

Please add an explicit comment that assumed pod is being modified by this function.

[wojtek-t]

Please add a comment that assume() is modifying the pod here (by setting the suggestHost to it).

[wojtek-t]

It's not the same - assume() method above is modifying the given pod.

[julia-stripe]

@wojtek-t I've added the comments you requested!

[bsalamat]

/lgtm

[julia-stripe]

/retest

[wojtek-t]

@julia-stripe - please squash commits and I will approve.

[julia-stripe]

done!

[julia-stripe]

/retest

[wojtek-t]

/lgtm

/retest

[k8s-merge-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, julia-stripe, wojtek-t

Associated issue: 49314

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• plugin/OWNERS [wojtek-t]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-merge-robot]

Automatic merge from submit-queue

[bsalamat]

Thanks a lot, @julia-stripe for debugging this issue and the fix!

[davidopp]

+1

[davidopp]

BTW I assume we should cherrypick this into 1.7, right?

[davidopp]

Ah, @dchen1107 had already marked this as cherypick-candidate.

[wojtek-t]

Yes - this will be cherrypicked to 1.7. I will take care of it.

[wojtek-t]

Cherrypick in #50240

[k8s-cherrypick-bot]

Commit found in the "release-1.7" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.