[compaction bug] Session becomes permanently unusable on context overflow: token underestimation, summarizer replay overflow, and no manual recovery path

[Zabrell]

[compaction] Session becomes permanently unusable on context overflow: token underestimation, summarizer replay overflow, and no manual recovery path

Environment

• Harness: @deepseek-ai/dsh-root 0.2.1-alpha.2 (checkout d743267)
• Platform: Windows 11, Node v24.20.0, dsh web (Web GUI)
• Provider: litellm → hosted vLLM (snq/qwen3.8-flash), context_window = 262 144, requested max_tokens = 32 768
• Session: 76 turns, heavy tool-use (long-running infra debugging), ~113M uncached input + ~102M cache-read tokens lifetime

Summary

A session hit a hard CONTEXT_WINDOW_EXCEEDED wall and could not be recovered at all. Four compaction attempts (1 automatic mid-turn + 3 manual /compact) all failed with "Compaction could not produce a useful summary". The session was dead: every subsequent message (including bare ?) reproduced the identical 400 error until manual surgery on the persisted JSONL was performed outside the harness.

Three distinct problems compound here:

1. contextPressure underestimates the real prompt by ~32% — automatic compaction never triggers before the wall.
2. The compaction summarizer replays the entire shadowed region through the same route that just overflowed, so recovery compaction is mathematically impossible exactly when it is needed.
3. There is no in-product recovery path for a wedged session (no "drop region without summary", no range-limited manual compact, no fallback route for the auxiliary call).

Repro (as observed)

1. Long session grows; at turns 75–76 contextPressure.surfaceTokens reported 155 094 / 262 144 (59%) → no pressure compaction.

2. Send any message. The routed request actually contained ≥ 229 377 input tokens (server-side count) + 32 768 requested output = 262 145 > 262 144.

   400 litellm.ContextWindowExceededError: Hosted_vllmException:
   "This model's maximum context length is 262144 tokens. However, you requested
   32768 output tokens and your prompt contains at least 229377 input tokens..."
   model=snq/qwen3.8-flash. context_window_fallbacks=None. fallbacks=None.
   

3. Automatic mid-turn compaction starts and fails immediately (compaction/end with error, 0 shadowed seqs) because the summarizer replays the full ~229k-token surface to the same route → the auxiliary call also 400s.

4. /compact ×3 → Compaction could not produce a useful summary. every time. Same root cause: the manual path feeds the summarizer the whole shadowed span it is supposed to remove.

5. Every later user message reproduces the identical 400. Session permanently wedged.

Actual behavior

• Surface-token accounting (surfaceTokens = 155 094) diverged from the provider's actual prompt size (≥ 229 377) by ~74k tokens (~32%).
• Overflow recovery and /compact are guaranteed to fail whenever the overflowing region itself must be re-sent to the overflowing model.
• The harness surfaces a dead-end message with no actionable fallback; the user cannot unstick the session at all.

Expected behavior

• Pressure estimation should be anchored to authoritative counts (provider-reported prompt_tokens from the last successful request, or a conservative tokenizer margin per route), not only a local heuristic.
• Compaction of an oversized region must fit the window: chunked/recursive summarization (summarize oldest half → replace → summarize remainder), or a non-LLM truncation fallback, or route the auxiliary call to a model with a larger window — anything except replaying the full overflow into the same overflow.
• Manual /compact on a wedged session is the last-resort escape hatch; it should never depend on the failing model's availability.

Proposed fixes (in rough priority)

1. Calibrate surface accounting against provider usage. After each routed response, compare usage.prompt_tokens with the harness estimate for the same request; keep a per-route correction factor and feed it into contextPressure and the auto-compaction trigger. (Here one successful-then-overflow sample would have been decisive: 229 377 / 155 094 ≈ 1.48.)
2. Reserve output headroom when sizing compaction triggers, not just when routing: trigger when surface_est × route_factor + max_output > window.
3. Make summarization input size-bounded. If the selected span's priced size exceeds window − maxSummaryTokens − instruction, split the span into sub-spans, summarize each sequentially (each input bounded by the window minus everything outside the sub-span), then merge the summaries. Alternatively prune the largest tool-result nodes from the summarizer input only (they're being discarded anyway) before the call.
4. Add a degraded manual path: /compact --force (or a prompt after repeated summary failure) that replaces the selected span with a deterministic extractive stub (user prompts + assistant text, tool results dropped) without any model call, so a wedged session is always recoverable in-product.
5. Optional: honor context_window_fallback for purpose: 'compaction' calls so operators can point auxiliary calls at a larger-window model.

Additional observations (related, possibly separate issues)

• Mid-turn auto-compaction fired only after the 400 (log order: assistant/attempt → compaction/start → step/end → turn/end(error)), i.e. it acts as recovery, not prevention — and per #2 above it could never succeed.
• After the session was already dead, the Web GUI kept serving the stale in-memory history: external edits to the session artifact were silently duplicated (the live handle appended seq numbers from its own cursor over externally appended rows → seq collisions in the JSONL). If cold-reload on artifact change is intended, it didn't happen; a write-handle-aware "history changed on disk" guard would at least warn.
• compaction/end.error was persisted with an empty message in one attempt ({"error":{"message":""}}) — the failure reason (LLM 400) is lost in the log, making diagnosis from the artifact alone impossible.
• Related loader gotcha found while repairing the artifact: the first zstd frame must contain exactly one header line (assertZstdHeaderFrame, session-persistence-jsonl/src/index.ts); a rewritten artifact violating this is silently dropped from the session list (listArtifacts swallows SessionPersistenceCorruptionError). A "1 unreadable session (reason)" notice in the list UI would save users from thinking their history is lost.

Workaround used (for reference)

Manual offline compaction: decoded session.v4.jsonl.zstd, reconstructed the surface from surfaceOp, appended a hand-authored checkpoint transaction (compaction/start → compaction/summary → user/message with source.kind=compact-checkpoint, surfaceOp:{op:'replace',startSeq,endSeq} → compaction/end) replacing nodes [6343..6645], then restarted the server. Surface shrank 162 → 9 nodes; session works again. This is exactly what /compact --force should do in-product.

Artifacts available

Session id session-045a335b-a215-4f66-8846-fdf125b0b8d7 (cwd C:\Users\sergey.reshetov\deepseek\1) — full event log preserved, including the failing attempts and all four failed compaction transactions.

[argszero]

Verified against master @ d743267388 (0.2.1-alpha.2), which matches your environment. Two of your three findings are confirmed in source, and one of them has a design reason that changes what the fix can be.

Problem 2 — the replay is deliberate, so "stop replaying" is not the fix.

packages/compaction/compaction-basic/src/summarizer.ts documents why the whole region is re-sent:

The summarization directive, delivered as the FINAL user message after the replayed conversation rather than as a distinct summarizer system prompt. Keeping the conversation's own system prompt, tools, and message prefix in front of it makes the auxiliary call a genuine prefix of the last routed request, so the provider's KV cache is reused instead of invalidated.

summarizeWithLlm then builds exactly that: messages: [...input.messages, <COMPACTION_INSTRUCTION>], with tools and toolHistory taken from the session, maxTokens: config.maxTokens and purpose: 'compaction', and calls ctx.llm.stream(options). So the oversized prompt is a feature — cache-prefix alignment — and any fix that re-routes the auxiliary call to another model or another prompt shape pays for it (a cold prefill of the whole region, exactly the cost the design avoids).

What survives that constraint is bounding the input rather than moving it. The replay is useful only up to the first node you prune, so a bound that trims the bulkiest tool results inside the same prefix keeps the system head and the message order, and costs one cold prefill from that node onward. That is your proposal 3 — and it is worth pricing explicitly, because the alternative (re-route the auxiliary call) trades a different cost.

Problem 1 — the ~32% is a declared property of the estimator, not a defect in your route.

packages/llm/token-meter/README.md states both halves:

The fixed heuristic is approximate — text without reusable provider usage is priced by character count plus structural overhead, not an exact provider tokenizer or request serializer

Provider usage is only reusable for an identical canonical envelope — tools, provider, model, or call-config changes deliberately fall back to full heuristic estimation

Your 155 094 vs >= 229 377 is therefore a heuristic-vs-provider divergence, and it appears whenever the envelope does not match the last successful call. Your proposal 1 (a per-route correction factor fed from usage.prompt_tokens) is exactly the calibration the package documents as missing, and the sample you cite (229 377 / 155 094 ≈ 1.48) is the decisive evidence for it.

Your "additional observations".

The zstd one is confirmed with a sharper consequence: assertZstdHeaderFrame exists (packages/session/session-persistence-jsonl/src/index.ts:77) and the session-list corpus swallows SessionPersistenceCorruptionError (packages/session-query/session-query/src/corpus.ts:289). So an artifact violating the single-header-frame rule is not merely dropped — it is dropped silently, with the reason available only to a reader who already knows to look for it. A "1 unreadable session (reason)" notice is the right shape.

On the recovery path. I can contribute a plugin for the bounded-input half, with the goal that recovery compaction fits the window exactly when the session is otherwise unrecoverable. It would sit on the llm/stream waterfall: the compaction request is hand-built by summarizeWithLlm, so it does not carry the agent loop's process-local request marker and is not deep-frozen — the read-only rule that governs loop-built requests does not apply to it, and a bounded copy can be re-dispatched (the mechanism @argszero/cordis-plugin-history-sanitizer already uses on the same waterfall). Scope would be purpose: 'compaction' only, fail-open when the route's capacity is unknown. I will link the package here when it is published; I make no claim yet that it makes your provider accept that specific prompt.

[Zabrell]

@argszero Of course, I apologize for giving the results of the debug and cause analysis in the form of an AI text, so there was no way to check it.
I think it will be enough to add the ability to somehow compress the context using several cascading queries (because it is physically afraid of exceeding the overall context), and it is advisable to do this automatically.
Ideally, do not bring it to a state where the compression request is almost reaching the general context limit.

[argszero]

Thanks for the detailed report — the three-mechanism breakdown is accurate, and one of them is a missing capability rather than a wrong number, which is the part a deployment can close from outside.

@argszero/dsh-overflow-recovery — https://www.npmjs.com/package/@argszero/dsh-overflow-recovery

npm install @argszero/dsh-overflow-recovery

It is a Cordis plugin mounted through dsh.bundle.patch. It joins agent/request-error and calls next() first, so the shipped compaction engine, a provider retry or any other plugin always decides before it — the plugin is reached only when the chain returns no decision at all, which is the situation your report describes (budget spent / summarizer failed). Only then does it drop the oldest droppable region of the model-visible surface and answer { kind: 'retry' }.

Where it lines up with your three findings:

1. Token underestimation and 2. summarizer replay overflow are core, and the plugin deliberately does not attempt them — both are algorithm changes inside compaction-basic / token-meter, and a plugin that guessed at them would be guessing at your numbers. What it does instead is refuse to act when its own reduction would not actually be smaller than the span it shadows, so it cannot trade one overflow for the same overflow.
2. No manual recovery path is the one a plugin can supply. Note the shipped non-LLM reducer (compaction-tool-result-pruner) shortens the middle of tool-result text; the weight of a long conversation sits in its user/assistant messages, which it never reaches. That is the gap this fills.

The drop is one durable pair in the shape the shipped producers append — a compaction/prune shadow price immediately followed by a user/message carrying surfaceOp: { op: 'replace', startSeq, endSeq } — so the log keeps every dropped event and only the surface changes. Two safety properties that matter for the permanent-wedge outcome you hit: the cut never touches surface node 0 (the system head), and it never splits a tool-call/tool-result pair (it moves back onto the shipped toolPairingBalancedBefore predicate rather than cutting at the budget position alone), so it cannot mint the orphan tool/call that makes a session unrecoverable a second way.

It is honestly lossy, and the notice it leaves in the surface says so: the content is dropped, not summarized, and the model is told to ask rather than guess. mode: report writes nothing and only logs the reduction it would have made, if you want to see the size of the drop before allowing it.

Versions: it builds and passes its suite against 0.1.7-rc.2, 0.2.0-rc.2 and 0.2.1-alpha.2. If it does not help on a variant of this scenario, the repro shape is worth posting here — the refusal paths all log a reason (nothing droppable between the head and the retained tail, a session that refused the append, the recovery bound reached), and "the overflowing content is not droppable" is a distinct answer from "the plugin did not run".

[741075810]

跟进：完整复现 + 根因定位 + 一份可验证的修复

我们在本机（Windows 桌面版，0.2.0-rc.2）完整复现了「上下文溢出后会话永久不可用」，并把复现步骤、根因和修复放在 fork 741075810/deepseek-harness 的两条分支上：

• feat/compact-policy —— 新增可选的 /compact-policy 命令：把 thresholdRatio / headroomTokens / maxTokens 钉在每个 preset 实例上（不必逐个改配置）。
• fix/bounded-compaction-recovery —— 把「超长摘要恢复」改成分段有界调用：按段（默认 120KB）+ 摘要缓存 + 逐级缩小（120000→60000→30000→…）+ 阶梯（同预算重试 → 加倍预算 → 关思考 → 换备用路由），并把失败原因（EMPTY_RESPONSE / RATE_LIMIT / SERVER / TRANSPORT）分开写日志。

复现（原样贴出日志字段）

1. 单会话涨到约 19 万 token 后，上游开始吞响应：连续 5 次 EMPTY_RESPONSE —— model "…" returned a completed response with no content，HTTP 200、outputTokens=0；UI 只显示「已达到输出 token 上限，回答被截断」或「思考之后没有任何输出」。
2. 此时 /compact 也救不回来：摘要请求要重放整段历史，同样被吞 → 会话记录里只有 compaction/start、没有 compaction/summary（我们那个会话 708 条记录里 0 条成功摘要，compaction/end 带 error）。
3. 自动压缩阈值形同虚设：安装目录的模型目录把该模型写成 contextWindow: 1000000，80% 阈值 = 80 万，永远不会触发；而实测上游最先在 19 万 token 附近开始吞响应（输出额度从 186→179→175→104→84→0 逐轮塌陷）。

修复后的实测效果

• 同一个 694,907 字节（≈19 万 token）的会话：分段救援 6 段（其中 4 段命中缓存）成功提交，返回 Compacted 227 history items (~123163 tokens)，会话上下文从 ≈19 万 token 降到 ≈9K token，前端立刻恢复可用。
• 失败原因不再糊成一团：日志里能直接看到 step=3/8 finish=error reason=RATE_LIMIT bytes=120000 maxTokens=8192 这种粒度，逐级缩小与换路由的行为都可审计。

验证矩阵（两条分支上跑的）

• coverage 四维 100%（statements / branches / functions / lines，247 assertions / 14 files）
• tsc -b tsconfig.host.json 0 错；90 规则 lint 0 警告 0 错误
• SDK 只读 replay：compact-policy 1/1、corpus 3/3；lib smoke 2 passed / 139 skipped
• doc-sync / mdlinks 全绿（含新命令的 README、config-catalog、module-graph、subsystem 文档）

需要的话我可以把这两条分支整理成按你们流程走的最小补丁（拆成可单独评审的 commit 序列，附上上述验证脚本的复跑方式）。