Retry scheduling pods after errors more consistently in scheduler

[julia-stripe]

What this PR does / why we need it:

This fixes 2 places in the scheduler where pods can get stuck in Pending forever. In both these places, errors happen and sched.config.Error is not called afterwards. This is a problem because sched.config.Error is responsible for requeuing pods to retry scheduling when there are issues (see here), so if we don't call sched.config.Error then the pod will never get scheduled (unless the scheduler is restarted).

One of these (where it returns when ForgetPod fails instead of continuing and reporting an error) is a regression from this refactor, and with the old behavior the error was reported correctly. As far as I can tell changing the error handling in that refactor wasn't intentional.

When AssumePod fails there's never been an error reported but I think adding this will help the scheduler recover when something goes wrong instead of letting pods possibly never get scheduled.

This will help prevent issues like #49314 in the future.

Release note:

Fix incorrect retry logic in scheduler

[k8s-ci-robot]

Hi @julia-stripe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k82cn]

I think we need to FinisBinding before ForgetPod; if ForgetPod failed, the assumed pod maybe still in cache.

[wojtek-t]

Agree. There were two problems here. One that we were returning here, the second is what @k82cn pointed at.

[wojtek-t]

This relies on the fact that Error will check if pod was bounded in the meantime and if so will not add it back to the unscheduled pods set (otherwise it would lead to an infinite loop).

This is true now, but can you please add a comment about it.

[wojtek-t]

Agree. There were two problems here. One that we were returning here, the second is what @k82cn pointed at.

[julia-stripe]

Thanks for the review! Added a comment & moved FinishBinding back up after bind. (as a side note, these are my first contributions to Kubernetes and I'm extremely impressed by how quickly and carefully people review code)

PTAL @wojtek-t

[cblecker]

/ok-to-test

[wojtek-t]

/lgtm
/approve no-issue

[k8s-merge-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: julia-stripe, wojtek-t

Associated issue: 49314

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• plugin/pkg/scheduler/OWNERS [wojtek-t]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[k8s-merge-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-merge-robot]

Automatic merge from submit-queue

[wojtek-t]

Cherrypick in #50240

[k8s-cherrypick-bot]

Commit found in the "release-1.7" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.