Recurrent State Placement

Independent recurrent-state placement

Hybrid models can contain both an attention KV history and fixed-size recurrent R/S state. This feature allows those two allocations to use different placement policies. A typical low-VRAM setup keeps the growing attention history in host memory while retaining the fixed recurrent state on the accelerator.

Control

--recurrent-state-offload keeps supported recurrent state offloaded even when --no-kv-offload places attention KV on the host. The default is off, preserving the coarser inherited placement behavior.

This option does not shrink recurrent state and does not alter its precision. With MTP, rollback planes multiply the recurrent-state allocation; use capped MTP recurrent planes only when that separate allocation becomes material.

Note

Backend scope: The policy is backend-capability driven, not CUDA-only. It affects hybrid models with recurrent memory; it has no useful allocation to change for a pure transformer.

Branch implementations

Branch	Implementation
beellama/main	src/llama-memory-hybrid.cpp, src/llama-model.cpp
llama/main	src/llama-memory-hybrid.cpp, src/llama-model.cpp
llama/dev	src/llama-memory-hybrid.cpp, src/llama-model.cpp
moe-cache	src/llama-memory-hybrid.cpp, src/llama-model.cpp

Design history: PR 25 introduced this split with pinned host KV so storage and execution policy resolve independently.

Feature index - Pinned host KV