Adaptive KV Cache Streaming: Implementation Approach, Scope, and Common Infra

[RaymondHuang210129]

Adaptive KV Cache Streaming: Implementation Approach, Scope, and Common Infra

Hello community and developers,

I’d like to start a discussion about whether my Adaptive KV Cache Streaming approach could be a good fit for upstream llama.cpp.

Adaptive KV Cache Streaming works by dynamically streaming a portion of the KV cache for each full-attention layer into GPU VRAM on demand, rather than keeping the entire KV cache resident in VRAM at all times. This allows llama.cpp to manage a KV cache larger than what can physically fit in VRAM.

The goal is to relax one of the major limitations of consumer GPUs, where the usable context length is often strictly bounded by the amount of available VRAM.

I’ve implemented a proof of concept in my fork:
https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

The results so far are quite promising. As a baseline, I compared my implementation against stock llama.cpp with GGML_CUDA_ENABLE_UNIFIED_MEMORY enabled. Unified Memory allows llama.cpp to allocate beyond the available VRAM without immediately crashing, but once VRAM becomes heavily oversubscribed, decode performance drops sharply due to page thrashing.

Adaptive KV Cache Streaming is designed to avoid that behavior by managing KV cache movement explicitly. In my tests, it significantly extends the usable context length before decode performance becomes impractical.

On an RTX 5070 Ti 16 GB running Unsloth Qwen 3.8 27B UD-Q3-XL with Q8 K / Q4 V KV cache, I got the following prefill and decode performance comparison:

Mechanism

The VRAM pool used for KV cache is dynamically divided into two regions: a resident region and a shared streaming ring buffer.

When the context is short enough for the entire KV cache to fit in the pool, the ring buffer is disabled and the whole pool is used as resident storage. The KV cache for all full-attention layers remains resident in VRAM, with each layer receiving an equal share of the available space.

As the context grows beyond what the resident pool can hold, a portion of each full-attention layer's KV cache is evicted from VRAM. The freed VRAM is then repurposed as a shared ring buffer. The evicted portion, as well as newly allocated KV cache beyond the resident region, is streamed from system RAM into this buffer as needed.

Because layers are processed in a predictable sequential order during decoding, the streamed KV cache can also be prefetched in that same order. A prefetch can be issued as soon as a ring-buffer slot becomes available, allowing the transfer to overlap with computation on earlier layers.

The size of the ring buffer is determined by two competing factors:

• Prefetch lookahead. The ring buffer needs to be large enough to provide sufficient lookahead so that a layer's KV cache has already finished transferring by the time the GPU reaches that layer.
• PCIe bandwidth. The ring buffer cannot simply keep growing. Increasing its size requires evicting more resident KV cache, which means more data must be transferred over PCIe for every decoded token. Once PCIe bandwidth becomes the bottleneck, further enlarging the ring buffer no longer helps and can actually hurt performance.

Caveat

The current implementation is still experimental and has mainly been validated on a single NVIDIA GPU with one server slot.

Supporting multiple parallel slots is more complicated because the resident/ring-buffer partition is dynamically adjusted based on the active context length. Different slots may require different partition sizes at the same time.

My implementation of streaming path is currently CUDA-specific. Performance has primarily been characterized with Qwen3.8-27B, Flash Attention, so other models, and backends still need broader testing.

This approach also does not eliminate the memory-bandwidth limit. As more KV cache is streamed, PCIe bandwidth eventually becomes the bottleneck, so the benefit depends on the GPU, PCIe link, context length, and KV cache size.

Finally, the streamed KV cache is backed by pinned system memory, so extending the context beyond VRAM capacity trades additional host RAM and PCIe traffic for a larger usable context.

Discussion

Backend abstraction

My current implementation is CUDA-specific, but I believe the same mechanism could also work on other traditional discrete-GPU backends where device memory is separate from system memory.

If we want to extend this mechanism to other backends, I wonder whether it would make sense to introduce a common abstraction for managing a bounded device-memory pool, including the resident region, streaming ring buffer, and host-to-device prefetching.

Dynamically repurposing VRAM between inference phases

In my implementation, I further enlarge the ring buffer at long context lengths by releasing the prefill CUDA graph once the server enters the decode phase, since that graph is no longer needed during decoding. The freed VRAM can then be repurposed for KV cache pool that further lengthen the usable context before hitting the PCIe bandwith limit.

This optimization currently assumes a single slot, but I think the underlying idea could be useful more broadly for single-GPU deployments: different phases of inference have different VRAM requirements.

I wonder whether a more general VRAM arena / memory-pool mechanism, capable of dynamically repurposing VRAM between different uses and inference phases, could unlock additional optimization opportunities.

[aebrer]

I have been running this for days now for my Qwen 3.8 27b route, and loving it. Works exactly as expected, and I've even been able to build off it.

[giveen]

@RaymondHuang210129 I had a similar idea but you fixed a few things I hadnt thought about or couldnt solve. I'll probably adapt your fixes to my own, mainly the tg regression issues I had.
TheTom#326

[giveen]

@RaymondHuang210129 i updated mine to include support MTP and other models, I really suggest you submit your idea to main llama.cpp, you did incredible work, far better than mine, so I copied your work.

TheTom#357

[troed]

Voicing support for this as part of inclusion in llama.cpp. I've used Raymond's fork since it was released with Qwen 3.8 27B IQ4/UD-Q3 models on 16GB 5060Ti as a daily driver. The speed when the streaming pool is not used is exactly as normal and then I can use much larger KV cache size with a predictable and steady loss of TG/s instead of a hard limit.