feat: use llama.cpp slot state save/restore instead of text-based compression to eliminate re-prefill

[hwangtab]

Problem

When context compression is triggered (at contextPercentageThreshold), qwen-code sends a summarized conversation history to the llama-server. This summary is a different token sequence from the original, causing:

1. Full cache miss — the server's LCP (longest common prefix) matching finds no reusable cached state
2. Full re-prefill — the entire summarized context must be reprocessed from scratch
3. Wasted compute — on long sessions (50K+ tokens), this is a significant re-prefill cost (often 100+ seconds on local hardware)

The server's prompt cache becomes useless at exactly the moment it would be most valuable.

Proposed Solution

llama.cpp server already exposes a slot state save/restore API:

POST /slots/{id_slot}/save    — saves full slot state (KV cache + recurrent state) to a file
POST /slots/{id_slot}/restore — restores from that file and continues

Instead of text-based compression, qwen-code could:

1. Before context limit is reached → call /slots/0/save to snapshot the current model state
2. When compression would normally trigger → call /slots/0/restore + send only the new tokens since the snapshot
3. The server resumes from the exact model state — zero re-prefill of prior context

This would replace "summarize history → resend full text" with "snapshot state → append new tokens".

Why This Is Especially Valuable for Qwen3.6-series Models

Qwen3.6-35B-A3B (and similar hybrid recurrent-transformer models) maintain a fixed-size recurrent state that already encodes a compressed representation of all past context. This state is exactly what is saved/restored by the slot API.

• The recurrent state IS a lossless (for the model's learned compression) summary of past context
• Saving and restoring it is architecturally more faithful than text summarization
• llama.cpp PR #24785 recently improved recurrent state correctness during prompt cache operations, making this approach more reliable now

Current Workaround

Users working with local llama.cpp backends currently maintain external markdown files (PROGRESS.md, QWEN.md) to manually persist context across compression boundaries — effectively a manual implementation of what MemGPT/Letta does with external memory. The slot API approach would make this unnecessary for within-session continuity.

Implementation Notes

• llama.cpp slot save path is configurable (--slot-save-path)
• The saved state includes both KV cache and recurrent state
• State files are ~60-500 MiB depending on context length and model size
• qwen-code would need to track: (a) which slot is active, (b) the token position at snapshot time, (c) the file path of the saved state
• On reconnect or server restart, fall back to current text-based compression

Related

• llama.cpp PR #24785 — recurrent state shrink/expand correctness
• llama.cpp PR #16391 — prompt cache (server-side LCP matching, current behavior)
• Issue more full prompt reprocessing in recent update? #5736 — full prompt reprocessing with local backends

[qwen-code-ci-bot]

This proposal is related to the prompt cache invalidation problem discussed in:

• more full prompt reprocessing in recent update? #5736 — full prompt reprocessing with local llama.cpp backends. The triage there identifies multiple sources of prefix instability (dynamic system prompt, MCP tool injection, memory prefetch, microcompaction) that would also affect the slot save/restore approach, since any token-level change after restore would still require the server to process new tokens from the snapshot point.

The two issues share the same goal: avoiding costly re-prefill when working with local backends. #5736 focuses on stabilizing the prompt prefix for LCP cache matching, while this issue proposes bypassing prefix matching entirely via state snapshot/restore.

[qwen-code-ci-bot]

Triage

• Type: feature request
• Labels: type/feature-request, category/performance, scope/caching, priority/P2
• Next: Escalate to maintainer — backend-specific optimization decision

The problem is real: when context compression fires, the summarized history is a completely different token sequence, so any server-side prompt cache (llama.cpp LCP matching, Anthropic cache breakpoints, etc.) is invalidated. For long sessions on local hardware, that re-prefill cost is significant.

However, the proposed solution is tightly coupled to llama.cpp — its slot save/restore API is unique to that backend. Qwen Code has no existing llama.cpp integration code, and supports many backends (OpenAI-compatible, Anthropic, DashScope, etc.) that don't expose equivalent state management. This means:

• Building a backend-specific optimization path inside a backend-agnostic codebase
• State files are 60–500 MiB on disk, requiring filesystem management and lifecycle cleanup
• Fallback to current text compression is still needed for non-llama.cpp backends, server restarts, and reconnection scenarios
• Claude Code's CHANGELOG shows the industry direction is improving text-based compaction (reactive compaction, cache TTL fixes, sub-agent cache optimization) rather than backend-specific state snapshots

On approach: A more general solution might be to preserve the original token prefix through compression — e.g., keep the first N turns verbatim (which the server can cache) and only summarize the tail. This would benefit all backends with prompt caching, not just llama.cpp. Have you considered this path? It might cover most of the performance win with a fraction of the backend coupling.

For the Qwen3.6-35B-A3B recurrent-state use case specifically, the argument is compelling — the recurrent state is the compression. But that's a very narrow target.

Escalating to maintainers for a direction call before deeper exploration. 🏷️

---

Direction Assessment

After reviewing the codebase:

• The current compression pipeline (chatCompressionService.ts, ~834 lines) is already sophisticated — 3-tier threshold ladder, side-query summarization with input slimming, post-compact history assembly with file/image restoration, reactive overflow recovery, and an inflation guard.
• No llama.cpp-specific code exists anywhere in packages/core or packages/cli. The codebase is deliberately backend-agnostic.
• Adding slot save/restore would introduce the first backend-specific optimization path — a meaningful architectural precedent.
• The Qwen3.6 hybrid recurrent model argument is technically sound but targets a narrow deployment scenario (local llama.cpp + Qwen3.6-A3B specifically).

Verdict: Suggest a smaller, backend-agnostic alternative.

Instead of integrating llama.cpp's slot API, a prefix-preserving compression strategy would address the root cause (cache invalidation from token sequence changes) for all backends:

1. When compression triggers, keep the first K turns of history verbatim (below the cache-friendly threshold)
2. Only summarize the tail portion (recent turns that haven't been cached yet)
3. The server-side prompt cache retains the verbatim prefix — no re-prefill needed

This achieves ~80% of the performance benefit with zero backend coupling. The llama.cpp slot API could be layered on top later as an optional backend-specific optimization, but the core improvement should be general-purpose.

中文说明

问题是真实的：上下文压缩触发后，摘要化的历史是全新的 token 序列，导致服务端 prompt cache（llama.cpp LCP 匹配、Anthropic cache breakpoints 等）完全失效。长会话在本地硬件上的 re-prefill 代价很高。

但提议的方案与 llama.cpp 强耦合 — slot save/restore API 是该后端独有的。Qwen Code 目前没有 llama.cpp 集成代码，且支持多种后端（OpenAI 兼容、Anthropic、DashScope 等），它们都没有等价的状态管理接口。这意味着：

• 需要在后端无关的代码库中构建后端特定的优化路径
• 状态文件 60–500 MiB，需要文件系统管理和生命周期清理
• 非 llama.cpp 后端、服务器重启、重连场景仍需回退到当前文本压缩方案
• Claude Code 的 CHANGELOG 显示行业方向是改进文本压缩（reactive compaction、cache TTL 修复、sub-agent cache 优化），而非后端特定状态快照

方案建议： 更通用的做法是在压缩时保留原始 token 前缀 — 例如保留前 N 轮原始对话（服务端可缓存），只摘要尾部。这样所有支持 prompt caching 的后端都能受益，而不仅仅是 llama.cpp。是否考虑过这个方向？可能用很小的复杂度覆盖大部分场景。

Qwen3.6-35B-A3B 循环状态的场景确实有说服力 — 循环状态本身就是压缩。但这是一个非常窄的目标。

已升级给维护者做方向判断。

---

方向评估

审查代码库后：

• 当前压缩管线（chatCompressionService.ts，~834 行）已经相当成熟 — 三级阈值阶梯、带输入精简的 side-query 摘要、带文件/图片恢复的压缩后历史组装、反应式溢出恢复、膨胀保护。
• packages/core 和 packages/cli 中没有任何 llama.cpp 特定代码。 代码库刻意保持后端无关。
• 添加 slot save/restore 将引入第一个后端特定优化路径 — 一个有重大意义的架构先例。
• Qwen3.6 混合循环模型的技术论点是合理的，但目标部署场景很窄（仅限本地 llama.cpp + Qwen3.6-A3B）。

结论：建议更小的、后端无关的替代方案。

与其集成 llama.cpp 的 slot API，不如采用前缀保留压缩策略来解决根因（token 序列变化导致的缓存失效），适用于所有后端：

1. 压缩触发时，保留前 K 轮原始历史（低于缓存友好阈值）
2. 只摘要尾部（尚未被缓存的近期轮次）
3. 服务端 prompt cache 保留原始前缀 — 无需 re-prefill

这能以零后端耦合实现约 80% 的性能收益。llama.cpp slot API 之后可以作为可选的后端特定优化叠加上去，但核心改进应该是通用的。

— Qwen Code · qwen3.7-max