|
On an A10 (24GB) serving a 7B model, is there a real throughput gain going from 0.85 to 0.95 gpu-memory-utilization, or just more KV cache headroom? |
Answered by
iacker
Jul 10, 2026
Replies: 1 comment
|
The gain is indirect: higher utilization grows the KV cache, which raises the number of sequences vLLM can batch concurrently. On a 7B/A10 you typically go from ~0.85 leaving room for a handful of extra concurrent sequences; throughput rises only until you become compute-bound, after which extra KV space does nothing. Rule of thumb: push it up until you see OOM risk under peak concurrency, then back off 0.02-0.03 for safety margin. |
0 replies
Answer selected by
iacker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The gain is indirect: higher utilization grows the KV cache, which raises the number of sequences vLLM can batch concurrently. On a 7B/A10 you typically go from ~0.85 leaving room for a handful of extra concurrent sequences; throughput rises only until you become compute-bound, after which extra KV space does nothing. Rule of thumb: push it up until you see OOM risk under peak concurrency, then back off 0.02-0.03 for safety margin.