Idea
Prototype a vLLM experiment based on RDKV: jointly choose KV-cache eviction and quantization under a memory budget rather than treating KV compression and eviction as separate policies.
Target Hardware
- DGX Spark / GB10 / sm121
- Strix Halo / RDNA 3.5 / gfx1151
Sources
- https://arxiv.org/abs/2605.08317v1 (RDKV: joint eviction + quantization under budget)
- https://arxiv.org/abs/2608.05326 (QEvict: three-tier recoverable eviction — full-precision hot windows, a quantized recoverable tier with dequantize-and-promote on attention reactivation, deletion only for lowest-confidence windows; motivated by attention-importance drift during decode. 2026-08-07 scan: mechanism genuinely absent — fork eviction is positional/structural only,
vllm/v1/core/kv_cache_manager.py:593-625, with no score-driven policy and no recoverable tier. Watch; the decode-time attention-score tap is the hard part since fused decode kernels do not expose per-block scores.)
Why It Might Matter
Both GB10 and Strix Halo are memory-capacity and memory-bandwidth constrained for long-context decode. A joint eviction/quantization policy could reduce KV memory footprint and bandwidth while preserving more useful context than uniform KV quantization or simple eviction heuristics.
Possible vLLM Work
- Read the RDKV algorithm and identify the smallest integration point in vLLM's KV cache manager or scheduler.
- Start with an offline trace/replay experiment over existing request/KV metadata before touching hot kernels.
- Compare against existing KV quantization/eviction behavior on long-context decode workloads.
- Gate any prototype behind a clear experimental flag and measure accuracy/perplexity impact.
Duplicate Check
Searched randomvariable/vllm issues by title keywords, arXiv ID 2605.08317, RDKV, and the stable paper title phrase; no existing equivalent issue was found.
Idea
Prototype a vLLM experiment based on RDKV: jointly choose KV-cache eviction and quantization under a memory budget rather than treating KV compression and eviction as separate policies.
Target Hardware
Sources
vllm/v1/core/kv_cache_manager.py:593-625, with no score-driven policy and no recoverable tier. Watch; the decode-time attention-score tap is the hard part since fused decode kernels do not expose per-block scores.)Why It Might Matter
Both GB10 and Strix Halo are memory-capacity and memory-bandwidth constrained for long-context decode. A joint eviction/quantization policy could reduce KV memory footprint and bandwidth while preserving more useful context than uniform KV quantization or simple eviction heuristics.
Possible vLLM Work
Duplicate Check
Searched randomvariable/vllm issues by title keywords, arXiv ID
2605.08317,RDKV, and the stable paper title phrase; no existing equivalent issue was found.