Idea
Evaluate regime-aware KV-cache scheduling ideas for vLLM's batched serving path under hard KV memory budgets and unknown response lengths.
Target Hardware
- DGX Spark / GB10 / sm121
- Strix Halo / RDNA 3.5 / gfx1151
Sources
Why It Might Matter
On smaller single-node targets, KV cache pressure can dominate scheduling quality. A scheduler that routes/adapts by prompt length, expected decode regime, and memory budget may improve throughput and tail latency compared with fixed heuristics when response lengths are unknown.
Possible vLLM Work
- Extract the paper's scheduling policy into a simulator using vLLM request traces.
- Compare against current vLLM scheduling on mixed prompt/decode workloads with constrained KV cache.
- Identify whether the policy needs only scheduler changes or also KV block-manager hints.
- Benchmark tokens/sec, admission latency, eviction/recompute behavior, and p95/p99 latency.
Duplicate Check
Searched randomvariable/vllm issues by title keywords, arXiv ID 2607.09248, regime-aware routing, and the stable paper title phrase; no existing equivalent issue was found.
Idea
Evaluate regime-aware KV-cache scheduling ideas for vLLM's batched serving path under hard KV memory budgets and unknown response lengths.
Target Hardware
Sources
Why It Might Matter
On smaller single-node targets, KV cache pressure can dominate scheduling quality. A scheduler that routes/adapts by prompt length, expected decode regime, and memory budget may improve throughput and tail latency compared with fixed heuristics when response lengths are unknown.
Possible vLLM Work
Duplicate Check
Searched randomvariable/vllm issues by title keywords, arXiv ID
2607.09248,regime-aware routing, and the stable paper title phrase; no existing equivalent issue was found.