Sharing prompt prefixes across sessions in llama-server

[rochaudhari]

Problem

Agent clients start every session with the same long prefix: the tool schemas and the system prompt. For Hermes, this prefix is 14.1K tokens. llama-server can only reuse the cache of the slot a request lands in. This causes three problems.

1. Every new session computes the prefix again. On Qwen3.8-27B, Hermes's default local model, this takes 10 seconds, or up to 13 seconds while other sessions run.
2. Saved prefixes get lost:
   • Idle slots are cleared when a new task starts.
   • The RAM cache skips a saved entry when the shared prefix is less than 25% of that saved conversation. --cache-ram restores a copy of the prefix, but loading the entry removes it from the cache, so the next session cannot use it.
   • Hybrid models such as Qwen3.8 can only resume from a saved checkpoint, and there is rarely one at the exact point where two prompts start to differ.
3. Every active session keeps its own copy of the prefix. This wastes KV memory and slows every decode step, because the unified cache scans the cells of all sessions.

Proposal

I propose three changes to llama-server. All three are opt-in. The core llama.cpp library does not change.

1. Cross-slot attach (--slot-attach). A new session takes the shared prefix from another slot with seq_cp, which shares the KV cells instead of copying them. Hybrid models attach at one of the other slot's checkpoints.
2. RAM prefix entries (--cache-prefix). The server saves the state at the first user message as a separate entry in the RAM prompt cache. The entry is copied when loaded, so it stays for later sessions. The 25% rule does not apply to it, and when the cache is full, the least recently used entry is removed first.
3. Divergence checkpoints for hybrid models. When a prompt matches the cached state only up to a point before the first user message, the server saves a checkpoint there. Later sessions can then resume exactly where prompts differ, for example at a date at the end of the system prompt.

Attach needs a unified KV cache and turns off with context shift. All three features turn themselves off on sliding-window models and DeepSeek-V4, and for requests that use speculative decoding, LoRA or images. The server refuses --cache-reuse while attach is on.

amosWeiskopf's comment in #21961 discusses the same core idea as attach: running slots share a prefix's KV cells through seq_cp. That work was tested on attention-only models. My proposal adds four things on top of it:

1. Hybrid models. Attach knows about checkpoints, and divergence checkpoints let sessions resume where prompts differ.
2. Prefixes that outlive sessions. RAM prefix entries keep the prefix after a session ends.
3. A real agent workload. Hermes's actual tool schemas and system prompt, on a dense hybrid, a hybrid MoE and an attention-only MoE.
4. Correctness checks. Outputs at every position, compared with a cold run and with today's cache reuse.

Results

Setup:
NVIDIA A40 (46 GB), CUDA 12.8, Hermes's tools and system prompt (~14K-token prefix), q8_0 KV, 4 slots, 3 sessions, MTP off.

	Qwen3.8-27B (dense hybrid, Hermes's default)	Qwen3.6-35B-A3B (hybrid MoE)	Qwen3-30B-A3B (attention-only MoE)
Hermes's prompts as they are
Time to first token, 3rd concurrent session	12.9 s -> 1.2 s	3.8 s -> 0.5 s	6.5 s -> 4.9 s
Time to first token, sessions on later days (from RAM)	10.0 s -> 2.6 s, then 1.25 s	2.8 s -> 0.9 s, then 0.6 s	3.0 s -> 2.9 s
One prefix shared by all sessions
Time to first token, 3rd concurrent session	12.9 s -> 0.52 s	3.9 s -> 0.26 s	6.6 s -> 0.11 s
KV memory, from no history to 32K tokens each	-66% to -23% (0.92 GiB)	-65% to -23% (0.29 GiB)	-65% to -23% (1.35 GiB)
Decode speed, from no history to 32K tokens each	+17% to +12%	+17% to +12%	+65% to +23%

With one shared prefix and no history, I grew Hermes's prompt with more project context and measured the 3rd session's time to first token and the decode speed-up:

Shared prefix	Qwen3.8-27B	Qwen3.6-35B-A3B	Qwen3-30B-A3B
14K tokens	12.9 s -> 0.52 s, decode +17%	3.9 s -> 0.26 s, decode +17%	6.6 s -> 0.11 s, decode +65%
24K tokens	25.4 s -> 0.57 s, decode +29%	7.6 s -> 0.29 s, decode +28%	16.3 s -> 0.14 s, decode +92%
32K tokens	37.6 s -> 0.59 s, decode +36%	11.6 s -> 0.30 s, decode +35%	27.4 s -> 0.17 s, decode +106%

Without sharing, the wait grows faster than the prefix does. With attach, the wait stays about the same as the prefix grows.

1. The prompt layout limits how much can be shared. Sharing stops at the first token that differs, and Hermes puts the date and, if enabled, a session ID at the end of its system prompt. The Qwen3.6 and Qwen3.8 templates put the tools first, so almost the whole prefix is still shared, but the Qwen3 template puts them after the system prompt, so only 3.5K of the 14K tokens are.
2. Outputs change about as much as they do with today's cache reuse. With attach, the top-1 token matches a cold run at 93–99% of positions, against 94–99% for today's prompt-cache reuse, and the other positions are near-ties. Sessions sharing a slot's cells disturb that slot no more than unrelated sessions running next to it: on Qwen3.8-27B, its 128 output tokens were the same as when it ran alone.

Questions

1. Would a first PR with only cross-slot attach be welcome? It would be server-only and off by default: seq_cp sharing between running slots, plus divergence checkpoints for hybrid models. It could use a single flag instead of three.
2. Should RAM prefix entries wait? They could be part of this work, or they could wait for the block-hash prompt cache that ngxson mentioned in server : add --cache-disk for a persistent prompt cache #28092.
3. Is it acceptable for attach to refuse --cache-reuse for now? Attach also turns off with context shift, because both move shared cells. The alternative is to add copy-on-write for shared cells to the library first.

The prototype works on current master. I am happy to open a PR in whatever scope you prefer, including a smaller one.