[Bug] Compaction request misses provider prefix cache entirely — summarizer doesn't inherit reasoningEffort / adds maxTokens

[feiniaoxzh]

[Bug Report] Compaction request misses the provider prefix cache entirely — summarizer doesn't inherit reasoningEffort / adds maxTokens

Category: Ideas (bug report)

Summary

When @deepseek-ai/dsh-compaction-basic triggers a compaction (auto compactIfNeeded or manual /compact), the summarizer LLM call is sent with different request parameters than normal agent turns. Under providers with prefix caching (e.g. the opencode-go gateway for deepseek-v4-flash), that cache-key divergence makes the compaction request itself miss the cache completely, re-billing the entire context (~100k+ tokens) at full price.

Evidence (real session data, opencode-go / deepseek-v4-flash / reasoningEffort: high)

Before fix — compaction request usage:

usage: {"inputTokens":122558,"outputTokens":1584}   // no cacheReadTokens → 100% miss

After locally patching summarizeWithLlm to inherit reasoningEffort and drop the extra maxTokens — next compaction:

usage: {"inputTokens":402,"outputTokens":3548,"cacheReadTokens":220928}  // 99.7% of prefix reused

The shadowed range was ~158k tokens; a full miss means ~122k tokens re-priced on every compaction. In a long-running session with frequent auto-compaction this is a substantial recurring cost.

Root cause

dsh-compaction-basic/lib/index.js, summarizeWithLlm (~lines 267-316) builds the summarizer options as:

{ provider, model, messages, system, tools, maxTokens: config.maxTokens(8192), sessionId, purpose: "compaction", signal }

It picks provider/model from the last routed request header, but does not inherit the rest of the header config — notably reasoningEffort (we use "high") — and it adds maxTokens: 8192 that normal turns don't send (normal requests in dsh-agent-loop inherit header.config wholesale, no explicit maxTokens).

The compaction call is intentionally a genuine prefix of the last routed request (per the design comments, so the provider KV cache is reused) — but the differing reasoning_effort / max_tokens params split the provider-side cache key, defeating that intent.

Proposed fix

In summarizeWithLlm, inherit request parameters the same way dsh-agent-loop's buildRequest does: take agent.session.requestHeader() → header.config (provider/model/reasoningEffort/…), skip fields where header.adapterDefaults.reasoningEffort === true, and drop the explicit maxTokens override (let the adapter default apply, same as normal turns).

i.e. re-use the header config wholesale instead of cherry-picking provider/model only.

Impact / context

• Claude Code's compaction reuses identical cache-safe parameters for exactly this reason ("the only new tokens are the compaction prompt itself").
• With the fix applied, the compaction request hit 220,928 cached tokens and only billed 402 new input tokens in our test.
• This affects any provider/gateway whose cache key includes reasoning_effort/max_tokens (DeepSeek official KV cache, OpenAI-style gateways, etc.).

Happy to open a PR once contributions are enabled, or provide the patch diff.

[zoahdev]

Confirmed at source level and implemented as a cherry-pick-ready branch.

Source audit (master 47f9438)

packages/compaction/compaction-basic/src/summarizer.ts (summarizeWithLlm) did exactly what the report describes: it resolved only provider/model from agent.session.requestHeader()?.config, then forced maxTokens: config.maxTokens (default 8192). Every other request parameter from the routed header - notably reasoningEffort - was dropped, so the compaction call split the provider cache key and re-billed the whole prefix.

dsh-agent-loop's requestProposal already has the correct semantics: inherit header.config wholesale, delete reasoningEffort/maxTokens only when header.adapterDefaults marks them as adapter-materialized.

Patch: fix/compaction-inherit-header-config

https://github.com/zoahdev/deepseek-harness/tree/fix/compaction-inherit-header-config

• summarizeWithLlm now spreads the routed header config wholesale (same helper semantics as agent-loop's proposal, with adapter-defaulted fields dropped).
• The forced maxTokens: config.maxTokens is gone: the summarizer forwards maxTokens only when the routed header itself carried one, so the call envelope matches normal turns exactly.
• The resolved compaction target (explicit summarizationProvider/Model or model policy) still wins for provider/model.

Verification

• tsc -b tsconfig.host.json - clean
• pnpm vitest run packages/compaction/compaction-basic/tests - 124/124, including two new tests:
  • wholesale header inheritance (reasoningEffort: high + maxTokens: 4096 reach the adapter)
  • adapter-defaulted fields are dropped (explicit high falls back to the adapter's low default; maxTokens absent)

Your session numbers tell the story: 122,558 input tokens / 0 cache on the old path vs 402 new input tokens + 220,928 cacheReadTokens after the change - the compaction instruction is the only novel input, exactly as the design comments intend.

Thanks for the reproducible writeup - this one saves real money on every long session. Happy to open the PR once contributions are enabled.

[NovaRouteAI]

@zoahdev nice confirmation at source level. One more knob worth folding into the fix: when the summarizer drops the reasoningEffort inherited from the parent session, the compacted context changes shape, so the next turn pays a full re-cache even when the prefix bytes match.

We log effort-drift alongside cache-hit rate to catch that.

Would you take a regression spec covering effort inheritance as part of the same branch?

[feiniaoxzh]

Thanks @zoahdev for the source-level confirmation and the fix branch — the 124/124 tests with the two new coverage cases make it solid. Good to see the analysis independently validated. Looking forward to the PR once contributions are enabled.

[zoahdev]

Thanks - and nice catch on your side. Your session numbers (122,558 input tokens / 0 cache -> 402 new input tokens + 220,928 cacheReadTokens) are now recorded in the branch description and the dsh-docs patch list, so the PR can carry the production evidence alongside the tests. If you want, I can also add a reproduction note naming the opencode-go gateway setup so reviewers can verify the cache-key behavior without a long session.

[feiniaoxzh]

Sure, go ahead and add the reproduction note — happy to help reviewers verify without a long session. Thanks for folding the session numbers into the branch description and docs patch list.

[zoahdev]

Reproduction note added - reviewers can now verify without a long session:

https://github.com/zoahdev/dsh-docs/blob/main/docs/specs/repro-compaction-cache-miss.md

Key contents:

1. Setup: opencode-go gateway + deepseek-v4-flash + reasoningEffort: high; trigger compactIfNeeded or /compact.
2. Before: {"inputTokens":122558,"outputTokens":1584} - no cacheReadTokens, 100% miss.
3. After: {"inputTokens":402,"outputTokens":3548,"cacheReadTokens":220928} - 99.7% prefix reuse.
4. Verify path: record the compaction event's usage; the tell is inputTokens near the full context with cacheReadTokens absent (before) vs small inputTokens + large cacheReadTokens (after).

Also folded into the branch description on the fix branch. Thanks again for the session numbers - they turned a code-level fix into a cost story reviewers can check in one step.