Can reclaim enforce fair-share (not just deserved quota) between sibling queues?

[lasse-ii]

Version: v0.15.0

Setup. Hierarchical queues for a shared multi-user GPU cluster:

fly-root  (quota = cluster)
├── fly-priority   gpu quota 1000000        # "critical" workloads
└── fly-shared     gpu quota 0              # parent of all users
    ├── fly-alice   gpu quota Q, weight 1
    ├── fly-bob     gpu quota Q, weight 1
    └── …

This works really well for two of our goals (verified against a kwok-faked cluster):

• fly-priority's large quota lets it reclaim from anyone — critical jobs preempt everything, including a user's in-quota GPUs, because the priority queue and the users diverge at fly-root where fly-shared is over its quota of 0.
• A sibling (fly-bob) reclaims another user's above-quota GPUs but never their guaranteed Q — peers are protected up to their floor.

Question. Cross-queue reclaim only ever seems to recover a queue's deserved quota (GuaranteeDeservedQuota), never its computed fair share. Concretely: with fly-alice running the whole cluster and fly-bob idle, fly-bob reclaims only up to its quota Q and stops — even though its fair share (with two active queues, equal weight) is half the cluster. Bob reaches a fair split only as Alice's jobs finish and freed GPUs are re-allocated, not by preemption.

Is there a supported way to make an under-fair-share queue reclaim from over-fair-share siblings immediately — i.e. enforce the dynamic fair share via preemption rather than only the static quota? We saw a MaintainFairShare reclaim strategy referenced in the code/docs but couldn't get it to fire in this scenario. Is it configurable (scheduler config / per-queue), or is fair-share strictly an allocation-time concern by design?

Thanks!

[lasse-ii]

Moved to an issue for tracking: #1934