Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.13

[denkensk]

Cherry pick of #72558 on release-1.13.

#72558: add goroutine to move unschedulablepods to activeq regularly

What type of PR is this?
/kind bug
/sig scheduling
/priority important-longterm

What this PR does / why we need it:
The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are added to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time.

Which issue(s) this PR fixes:
Fixes #72122

Special notes for your reviewer:

Move unschedulable pods to activeq if they are not retried for more than 1 minute

[k8s-ci-robot]

Hi @denkensk. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Huang-Wei]

/ok-to-test

[Huang-Wei]

/lgtm

[denkensk]

/cc @bsalamat @Huang-Wei

[bsalamat]

/lgtm
/approve

Thanks, @denkensk!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, denkensk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [bsalamat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tpepper]

/retest

[tpepper]

pull-kubernetes-e2e-kops-aws likely will not pass due to k/k issue 73444

[k8s-ci-robot]

@denkensk: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kops-aws	06d31c3	link	/test pull-kubernetes-e2e-kops-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.