Independent Draft KV Residency

Independent draft KV GPU residency

A separately loaded speculative draft model owns a separate context and can benefit from a different KV placement than the target. This feature lets the draft inherit target placement by default or receive its own GPU-resident layer count.

Control

Use --spec-draft-kv-gpu-layers N (alias --kv-gpu-layers-draft N). Omitting it inherits the target policy. An explicit zero selects host placement for eligible independently owned draft KV even when the target keeps some layers on the GPU.

Shared KV follows the context that actually owns it. Integrated MTP uses the target model and does not create the same independent draft-model KV allocation, so this control is mainly for model-backed drafters.

Moving draft KV to the host can save enough VRAM for a larger target context, but a slow drafter can erase speculative gains. Measure end-to-end accepted output throughput rather than the draft model in isolation.

Branch implementations

Branch	Implementation
beellama/main	common/speculative.cpp, common/arg.cpp
llama/main	common/speculative.cpp, common/arg.cpp
llama/dev	common/speculative.cpp, common/arg.cpp
moe-cache	common/speculative.cpp, common/arg.cpp

Design history: PR 29. See upstream speculative decoding for normal modes and model requirements.

Feature index - Partial target KV residency