fix: requeue pods rejected by Extenders properly

[sanposhiho]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Extender doesn't support any kind of requeueing feature like EnqueueExtensions in the scheduling framework.

When Extender filters out some Nodes, we don't set any unschedulable plugins at all.
It means Extender is completely ignored during the requeueing process.

So, what's happening is:

• If Extender filters out all Nodes during scheduling, this Pod is soon retried because this Pod doesn't have any plugin name in unschedulable plugins. The scheduling queue doesn't put this Pod into the unschedulable pod pool, but put into activeQ/backoffQ.
• If Extender filters out some Nodes and plugins filter out all other Nodes, this Pod is retried based on plugins' QueueingHint. Even if some cluster events happen and they could change Extender's decision, this Pod won't be requeued to activeQ/backoffQ.

The latter case is serious because it could make Pods stuck in unschedulable pod pool in 5min in the worst case scenario.

This PR makes the scheduling queue aware of extenders' failures.
After this PR, when Extenders reject some Nodes and the pod ends up being unschedulable, this Pod will be requeued from unschedulable pod pool to activeQ/backoffQ by any kind of cluster events.

Which issue(s) this PR fixes:

Fixes #122019

Special notes for your reviewer:

Probably, we have to cherry-pick this PR into past versions?

Does this PR introduce a user-facing change?

The scheduling queue didn't notice any extenders' failures, it could miss some cluster events,
and it could end up Pods rejected by Extenders stuck in unschedulable pod pool in 5min in the worst-case scenario.
Now, the scheduling queue notices extenders' failures and requeue Pods rejected by Extenders appropriately.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.29 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.29.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Nov 23 09:57:18 UTC 2023.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sanposhiho]

/cc @alculquicondor

I think it's worth cherry-picking this PR in older versions.

[sanposhiho]

/hold

To involve other approvers

[alculquicondor]

/lgtm
/approve
how far do we need to cherry-pick?

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 33b5a3ba2a251a8b9a749a9185978a2e9ae1d34b

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [alculquicondor,sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sanposhiho]

how far do we need to cherry-pick?

Well, it's better to cherry-pick this for all supported versions because this event-based requeueing has been existing for a long time.

[alculquicondor]

sgtm

[sanposhiho]

/unhold

Got approval, not merged anyway though