You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Opt-in sparse-backed KV cache allocation (Vulkan / Linux CPU) to avoid upfront VRAM reservation
#32285
Opt-in sparse-backed KV cache allocation (Vulkan / Linux CPU) to avoid upfront VRAM reservation
Hi all,
Before opening a new PR, I wanted to check if there is interest in an opt-in, sparse-backed KV cache allocation approach, or if this should wait for the broader memory/context rework mentioned by ggerganov in #11577.
The idea
Instead of allocating the entire physical KV cache upfront when loading a model with a large -c, the KV cache acts like a dynamically expanding virtual disk: it reserves the full virtual address range at startup, and commits physical memory pages on demand as tokens are actually processed.
Unlike #21757, which starts small and reallocates/copies the buffer when it fills up, this approach does no resizing and no data copies:
Vulkan: Uses VK_BUFFER_CREATE_SPARSE_BINDING_BIT / VK_BUFFER_CREATE_SPARSE_RESIDENCY_BIT and binds physical memory pages via vkQueueBindSparse as the KV head advances. If physical memory runs out mid-generation, committing the next block fails cleanly, the request is rejected as if the context were full, and the server stays alive.
Linux CPU: Uses standard anonymous mmap with read/write permissions (Linux only). The OS assigns real physical memory to each page only when it is first written to. When the buffer is cleared, it uses madvise(MADV_DONTNEED) instead of writing zeros, which would otherwise make every page resident at startup.
Initial numbers
These figures were measured with the five commits on my fork (https://github.com/enriqxxx/llama.cpp/tree/kv-lazy), which also make the compute memory grow on demand along with the conversation cache (so the savings reflect both combined). Tested locally on an AMD APU with 128 GB unified memory running Ubuntu:
1B model @ 1M context: 8.24 GiB → 0.86 GiB at startup
4B model @ 262K context: 8.94 GiB → 4.41 GiB at startup
1B model, CPU-only @ 262K context: 2.87 GiB → 1.27 GiB process RSS at startup
Known limitations
Once the context window is completely filled, total memory usage matches standard upfront allocation.
Memory is not given back when a conversation ends or a slot is reset: once committed, it stays until the server restarts.
On CPU, if system memory runs out entirely, the OS OOM killer can terminate the process.
Proposed scope
Strictly opt-in (off by default).
Start with Vulkan first, followed by Linux CPU support.
Would a clean, self-contained PR with this scope be welcome, or does it conflict with the planned memory architecture refactor from #11577?
(Note: My previous PR #32170 was closed by am17an for skipping this step—holding off on code until getting feedback on the direction here. Full transparency: the core idea and direction are mine, the code was written with AI assistance (Claude), and I have thoroughly reviewed, tested, and run it in production.)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Opt-in sparse-backed KV cache allocation (Vulkan / Linux CPU) to avoid upfront VRAM reservation
Hi all,
Before opening a new PR, I wanted to check if there is interest in an opt-in, sparse-backed KV cache allocation approach, or if this should wait for the broader memory/context rework mentioned by ggerganov in #11577.
The idea
Instead of allocating the entire physical KV cache upfront when loading a model with a large
-c, the KV cache acts like a dynamically expanding virtual disk: it reserves the full virtual address range at startup, and commits physical memory pages on demand as tokens are actually processed.How this differs from #21757 (
--kv-dynamic)Unlike #21757, which starts small and reallocates/copies the buffer when it fills up, this approach does no resizing and no data copies:
VK_BUFFER_CREATE_SPARSE_BINDING_BIT/VK_BUFFER_CREATE_SPARSE_RESIDENCY_BITand binds physical memory pages viavkQueueBindSparseas the KV head advances. If physical memory runs out mid-generation, committing the next block fails cleanly, the request is rejected as if the context were full, and the server stays alive.mmapwith read/write permissions (Linux only). The OS assigns real physical memory to each page only when it is first written to. When the buffer is cleared, it usesmadvise(MADV_DONTNEED)instead of writing zeros, which would otherwise make every page resident at startup.Initial numbers
These figures were measured with the five commits on my fork (https://github.com/enriqxxx/llama.cpp/tree/kv-lazy), which also make the compute memory grow on demand along with the conversation cache (so the savings reflect both combined). Tested locally on an AMD APU with 128 GB unified memory running Ubuntu:
Known limitations
Proposed scope
Would a clean, self-contained PR with this scope be welcome, or does it conflict with the planned memory architecture refactor from #11577?
(Note: My previous PR #32170 was closed by am17an for skipping this step—holding off on code until getting feedback on the direction here. Full transparency: the core idea and direction are mine, the code was written with AI assistance (Claude), and I have thoroughly reviewed, tested, and run it in production.)
All reactions