Pinned Host KV

Pinned host KV

Pinned host KV keeps CPU-resident attention history in accelerator-visible host memory instead of ordinary pageable RAM. It is still system memory, not VRAM. The intended use is a long context that cannot fit its complete KV cache on the accelerator while attention operations remain offloaded.

Note

Backend scope: This is not a CUDA-only option. The selected backend and KV implementation must expose a suitable host buffer type. CUDA is the most extensively validated path. Unsupported layouts must fall back explicitly or reject the placement; parsing the flag alone does not prove support.

Control and behavior

Enable it with --no-kv-offload --kv-cpu-pinned. Leave --op-offload enabled if accelerator attention over host-resident KV is intended. --no-op-offload selects a different CPU-execution route.

Pinned pages can improve transfer behavior and make asynchronous copies possible, but they consume unswappable system memory. Deep-context attention still moves or reads context-linear history, so pinned storage is a capacity tool rather than a guarantee of higher decode speed.

The implementation separates persistent KV placement from attention execution placement. This distinction is shared by standard KV and the supported hybrid/SWA cache constructors. It composes with partial target residency, independent draft residency, and recurrent-state placement.

Branch implementations

Branch	Implementation
beellama/main	common/arg.cpp, src/llama-kv-cache.cpp, src/llama-model.cpp
llama/main	common/arg.cpp, src/llama-kv-cache.cpp, src/llama-model.cpp
llama/dev	src/llama-kv-cache.cpp, src/llama-context.cpp
moe-cache	src/llama-kv-cache.cpp, src/llama-context.cpp

Design history: PR 25, separate host storage placement. BeeLlama's earlier development record is summarized in its fork-owned CPU KV feature delta.

For ordinary KV options and cache types, use the upstream server documentation.

Feature index - 16 GB setup