|
Should I use Kueue for LLM batch inference jobs, or is HPA on a Deployment enough? |
Answered by
iacker
Jul 10, 2026
Replies: 1 comment
|
HPA is fine for a long-lived serving Deployment reacting to load. Kueue is the right tool when you have queued, quota-bound jobs competing for scarce GPUs — it does gang scheduling, fair-sharing across teams, and admission control so you do not oversubscribe. Serving = HPA; batch/campaign workloads on a fixed GPU pool = Kueue. They compose: HPA for the endpoint, Kueue for the batch lane. |
0 replies
Answer selected by
iacker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
HPA is fine for a long-lived serving Deployment reacting to load. Kueue is the right tool when you have queued, quota-bound jobs competing for scarce GPUs — it does gang scheduling, fair-sharing across teams, and admission control so you do not oversubscribe. Serving = HPA; batch/campaign workloads on a fixed GPU pool = Kueue. They compose: HPA for the endpoint, Kueue for the batch lane.