How does Kueue behave when GPUs fail but the node remains Ready in on-prem clusters?

[kimminw00]

I’m using Kueue in an on-prem GPU cluster.

Assume the following setup:

Guaranteed(standard) ClusterQueue:
- nominal GPU quota: 100

Opportunistic(besteffort) ClusterQueue:
- nominal GPU quota: 0
- currently using the 50 GPUs from the cohort shared nominal quota

Now suppose 8 GPUs fail at the hardware or device-plugin level:

Guaranteed entitlement: 100 GPUs
Available GPUs after failure: 92 GPUs
Opportunistic usage: 50 GPUs

The node itself remains Ready, but the device plugin reports the failed GPUs as unavailable, for example by reducing Node.status.allocatable["nvidia.com/gpu"].

From a policy perspective, I would expect the guaranteed queue to be able to reclaim capacity from opportunistic workloads, since opportunistic workloads are using borrowed GPUs.

My understanding is:

1. If a new or requeued guaranteed workload needs GPUs and cannot fit, Kueue may preempt opportunistic workloads in the same cohort, assuming reclaimWithinCohort allows it and the opportunistic workload is borrowing quota.

2. However, if the guaranteed workload is already admitted and 8 GPUs fail while the node stays Ready, it is unclear whether Kueue automatically triggers reclaim preemption.

3. TAS node-failure recovery seems to handle node-level failures, such as a failed or deleted node. A GPU-only failure on a Ready node may not enter the NodeFailures path.

4. The waitForPodsReady.recoveryTimeout value also seems to involve a trade-off: if it is too short, transient node or device issues may cause frequent evictions, which can be especially disruptive for large LLM workloads because recovery may involve pod rescheduling, model reloading, distributed worker restart, warmup, and readiness recovery; if it is too long, recovery from real GPU failures may be delayed.

So my question is:

Can GPU allocatable capacity loss on a Ready node trigger preemption of opportunistic workloads so that a guaranteed ClusterQueue can recover its nominal quota?

If not, is the expected approach to use custom on-prem health tooling to evict or requeue the affected guaranteed workload, so that it goes through admission again and can trigger normal Kueue preemption?

The desired flow would be:

GPU device failure detected
  ↓
Affected guaranteed workload is requeued
  ↓
Kueue re-runs admission
  ↓
Guaranteed workload needs its 100-GPU quota again
  ↓
Kueue preempts opportunistic borrowed workloads if needed
  ↓
Guaranteed workload is admitted again

Is this the right way, or is there a more Kueue-native way to handle resource-level failures such as GPU or NIC failures when the node itself remains Ready?

[amy]

Maybe try checking out TASReplaceNodeOnNodeTaints. Not entirely sure how this feature works, but you can test what happens if you add noSchedule taints on a node and see what happens.