Live Context Workspace

Live-context workspace

Live-context sizing reserves supported attention compute buffers from the padded live physical KV extent instead of the complete configured context at startup. It is useful when -c is much larger than the sequence currently in use.

Control and growth

Enable --live-context-workspace; the default is off. The reservation starts small and grows geometrically with live KV, up to configured capacity. Unsupported contexts keep full-context sizing and log why.

This changes transient workspace only. Model weights and persistent KV remain unchanged. It composes with phase-aware workspace and the automatic compact causal mask, but none of those controls substitutes for the others.

Note

Backend scope: Logical sizing is backend-generic. Physical tail release depends on allocator behavior. CUDA VMM can expose the clearest driver-level reduction; another backend may keep a reusable pool even after the logical plan shrinks.

Branch implementations

Branch	Implementation
beellama/main	src/llama-context.cpp, src/llama-memory.h
llama/dev	src/llama-context.cpp, src/llama-memory.h
moe-cache	src/llama-context.cpp, src/llama-memory.h

Use llama/dev when you need this experimental control; llama/main does not include it.

Design history: PR 8 for the BeeLlama line and PR 40 for llama/dev.

Feature index - llama/dev setup