add(nodevolumelimits): return UnschedulableAndUnresolvable when PVC is not found

[sanposhiho]

What type of PR is this?

/kind feature

What this PR does / why we need it:

When PVC is not found, currently, we return Error from the plugin.
When error is returned from the plugin, the scheduler keeps retrying this Pod's scheduling because the scheduler doesn't put pods failed by Error status into unschedQ.

This PR makes nodevolumelimits return UnschedulableAndUnresolvable when PVC is not found, and register PVC added event in EventsToRegister so that the scheduling queue retry the Pod only when PVC is created.
In the future, it'll be wiser via QHint as part of #118893 when to retry scheduling, but this PR doesn't do that for now.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The scheduler retries Pods, which are failed by nodevolumelimits due to not found PVCs, only when new PVCs are added.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sanposhiho]

/hold

To go thru two reviews.

[utam0k]

I'll take on implementing it as part of #118893.

[sanposhiho]

/hold

waiting for #121953

[sanposhiho]

/unhold

#121953 is merged.

[sanposhiho]

/hold

To go thru two reviews.

[sanposhiho]

@kubernetes/sig-scheduling-approvers

Sorry for rushing, can anyone take a review for this? It'll block #121508.

[ahg-g]

/assign @msau42

[ahg-g]

/lgtm

I think this makes sense, but it would be good to have @msau42 double check.

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 891a973a766cc81aeced0d606ef9f6e2ea3e37f2

[kerthcet]

My first thought is can we return Pending when PVC not found, it's similar to DRA waiting for allocation.

[alculquicondor]

/approve
/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/framework/plugins/nodevolumelimits/OWNERS [alculquicondor,sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sanposhiho]

@kerthcet You're correct. We can change all of this kind of not-found error to generate Pending state instead of UnschedulableAndUnresolvable - let's track that at #123227.