Automated cherry pick of #71551: activate unschedulable pods only if the node became more

[mlmhl]

Cherry pick of #71551 on release-1.11.

#71551: activate unschedulable pods only if the node became more

[mlmhl]

/retest

[mlmhl]

/assign @bsalamat

[mlmhl]

/test pull-kubernetes-bazel-test

[bsalamat]

/lgtm
/approve

Thanks, @mlmhl!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, mlmhl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [bsalamat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

[foxish]

It looks to me like this broke a storage test in https://k8s-testgrid.appspot.com/sig-release-1.11-blocking#gce-cos-1.11-serial

cc @mlmhl @bsalamat

[foxish]

Can someone from @kubernetes/sig-scheduling-misc confirm if this is indeed the cause? If not - I'll try and revert it in a couple hours.

[bsalamat]

@foxish What are your reasons to believe this PR broke the storage test? The storage test seems to be flaky even before the merge of this PR. And, this PR is an important scheduler fix. We need it in 1.11.

[foxish]

Based on testgrid it seems correlated - this test,

[sig-storage] PersistentVolumes-local Stress with local volume provisioner [Serial] should use be able to process many pods and reuse local volumes was not flaky.

It started to fail when c4240ec was committed and became green again when b1d75de (revert PR) was committted.

[bsalamat]

@foxish This fix has been back-ported to 1.12 and 1.13 as well (and of course exists in HEAD too). It is odd that the storage test depends on the internal algorithms of the scheduler only in 1.11. We should ask SIG Storage to fix their test.

[mlmhl]

I run this failed test in my local environment and found the reason is that:

When pods are created, the local volume PVs haven't been created yet, so all pods are marked as Unschedulable due to didn't find available persistent volumes to bind. This won't be a problem in previous version as scheduler will retry these Unschedulable pods(by invoking MoveAllToActiveQueue) when receiving node heartbeat updates.

However, after this PR cherry-picked, scheduler won't retry unschedulable pods anymore if nodes only updated for heartbeat, without any meaningful changes. So all pods are staying in pending state and finally timeout.

BTW, I found that this problem doesn't exist in 1.12 and 1.13, because from 1.12 scheduler will retry unschedulable pods also if any pvs added/updated. So I think we should cherry-pick #65616 to 1.11 first before we cherry-pick this PR. cc @bsalamat @msau42

[bsalamat]

Thanks a lot, @mlmhl for your investigation.

@msau42 @cofyc
Please cherry pick #65616 to 1.11 so that we can cherry pick this PR into 1.11 as well.

[msau42]

preparing cherry pick here: #72127