Partial Target KV Residency

Partial target KV GPU residency

Partial target residency keeps a selected number of independently owned attention-KV layers on their assigned accelerator devices while the remaining target KV layers use host storage. It is a layer-placement budget, not a token-recency window.

Control and accounting

Use --no-kv-offload --kv-gpu-layers N. N=0 leaves all supported attention-KV layers under the host policy. Raising N spends VRAM to reduce host traffic. The budget counts independently owned eligible layers; shared KV follows its real owner and is not double-counted.

The runtime resolves one placement plan and reuses it for allocation, capability checks, store planning, and precision tails. Specialized caches must decline a split they cannot represent.

Note

Backend scope: This is not CUDA-only. Actual device placement depends on the cache layout and backend buffer types. Verify the startup allocation report for the chosen model.

Branch implementations

Branch	Implementation
beellama/main	src/llama-kv-cache.cpp, src/llama-kv-cache-kvarn.cpp
llama/main	src/llama-kv-cache.cpp, src/llama-context.cpp
llama/dev	src/llama-kv-cache.cpp, src/llama-context.cpp
moe-cache	src/llama-kv-cache.cpp, src/llama-context.cpp

Design history: PR 28, partial GPU residency.

Start at zero, measure loaded and deep-context VRAM, then raise the count in small steps. The useful count depends on per-layer KV size and the device topology, not just the model's total layer count.

Feature index - Independent draft KV residency