[proposal] Introducing QueueingHint into the elasticquota plugin

[qinfustu]

What is your proposal:

The Kubernetes scheduling framework provides a QueueingHint mechanism. If this mechanism could be integrated into elastic quota management, it would help mitigate the issue where multiple resource quotas exist in a cluster. Specifically, when some quotas have a large number of high-priority pods stuck in a pending state, they may monopolize scheduling cycles and prevent pods in other quotas—even those with sufficient resources—from being scheduled.

​​Introduction to QueueingHint​​：

1. https://kubernetes.io/blog/2024/12/12/scheduler-queueinghint/
2. https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4247-queueinghint/README.md
3. https://cloud.tencent.com/developer/article/2393595

Why is this needed:

Is there a suggested solution, if so, please add it:

[googs1025]

/cc

[ZiMengSheng]

Good Suggestion! Maybe moving some quota check to preEnqueue is enough!

[qinfustu]

Good Suggestion! Maybe moving some quota check to preEnqueue is enough!

I understand that there are two aspects to reducing wasted scheduling cycles here:

1. QueueingHint enables precise control over which cluster events can trigger a pod to move from the unschedulableQ to the activeQ. If QueueingHint is properly implemented, it can significantly reduce ineffective scheduling. For example, in the elasticquota plugin, pod deletion events where no quota is configured can be ignored.
2. preEnqueue is used to determine which pods are eligible to enter the scheduling logic. Pods rejected during the preEnqueue phase will also rely on ClusterEventWithHint to trigger their re-entry into the preEnqueue phase.

[ZiMengSheng]

Good Suggestion! Maybe moving some quota check to preEnqueue is enough!

I understand that there are two aspects to reducing wasted scheduling cycles here:

1. QueueingHint enables precise control over which cluster events can trigger a pod to move from the unschedulableQ to the activeQ. If QueueingHint is properly implemented, it can significantly reduce ineffective scheduling. For example, in the elasticquota plugin, pod deletion events where no quota is configured can be ignored.
2. preEnqueue is used to determine which pods are eligible to enter the scheduling logic. Pods rejected during the preEnqueue phase will also rely on ClusterEventWithHint to trigger their re-entry into the preEnqueue phase.

Yes, we might need to implement the PreEnqueue interface and more precisely, EnqueueExtensions. EnqueueExtensions could even perform quota checking, but this might incur a significant performance overhead. Perhaps, as you suggest, simply excluding irrelevant events and reducing the frequency of irrelevant pods entering ActiveQ/BackoffQ is a viable approach with relatively low performance overhead.

[qinfustu]

Good Suggestion! Maybe moving some quota check to preEnqueue is enough!

I understand that there are two aspects to reducing wasted scheduling cycles here:

1. QueueingHint enables precise control over which cluster events can trigger a pod to move from the unschedulableQ to the activeQ. If QueueingHint is properly implemented, it can significantly reduce ineffective scheduling. For example, in the elasticquota plugin, pod deletion events where no quota is configured can be ignored.
2. preEnqueue is used to determine which pods are eligible to enter the scheduling logic. Pods rejected during the preEnqueue phase will also rely on ClusterEventWithHint to trigger their re-entry into the preEnqueue phase.

Yes, we might need to implement the PreEnqueue interface and more precisely, EnqueueExtensions. EnqueueExtensions could even perform quota checking, but this might incur a significant performance overhead. Perhaps, as you suggest, simply excluding irrelevant events and reducing the frequency of irrelevant pods entering ActiveQ/BackoffQ is a viable approach with relatively low performance overhead.

To address performance overhead concerns, could we consider enabling goroutines to handle quota checks? This would be feasible if the quota checking mechanism is thread-safe and supports concurrent verification. Additionally, it will not block the enqueue of pods that do not have a quota configured.

[ZiMengSheng]

Good Suggestion! Maybe moving some quota check to preEnqueue is enough!

I understand that there are two aspects to reducing wasted scheduling cycles here:

1. QueueingHint enables precise control over which cluster events can trigger a pod to move from the unschedulableQ to the activeQ. If QueueingHint is properly implemented, it can significantly reduce ineffective scheduling. For example, in the elasticquota plugin, pod deletion events where no quota is configured can be ignored.
2. preEnqueue is used to determine which pods are eligible to enter the scheduling logic. Pods rejected during the preEnqueue phase will also rely on ClusterEventWithHint to trigger their re-entry into the preEnqueue phase.

Yes, we might need to implement the PreEnqueue interface and more precisely, EnqueueExtensions. EnqueueExtensions could even perform quota checking, but this might incur a significant performance overhead. Perhaps, as you suggest, simply excluding irrelevant events and reducing the frequency of irrelevant pods entering ActiveQ/BackoffQ is a viable approach with relatively low performance overhead.

To address performance overhead concerns, could we consider enabling goroutines to handle quota checks? This would be feasible if the quota checking mechanism is thread-safe and supports concurrent verification. Additionally, it will not block the enqueue of pods that do not have a quota configured.

The overhead I'm talking about is the lock collision overhead after a seperate goroutine. But the overhead may not be that big, so let's test it later.

[qinfustu]

/assign