Auto preset: every tool call issues a full-context review request — 42% of a 13-day bill

[cinast]

Environment: DSH Desktop 0.2.0-rc.2 (web) on Windows 11, provider deepseek-account / deepseek-flash, permission preset Auto. Requests were observed through a local proxy (billion-context 0.1.211) in front of api.deepseek.com, so every upstream request is attributable; no DSH-internal instrumentation was used.

Related: #11834 (seeded first-turn re-upload) — same observing setup, different finding.

What was measured

Under the Auto preset the reviewer is a separate LLM route call. The proxy namespaces lanes by system-prompt fingerprint, so the reviewer's requests land on their own lane per conversation (<session-id>|sub:1ee45271ed060930). It is not a special-cased ID: the proxy simply sees a second, distinct instructions string in that session and appends its hash; the string happens to be constant, so every reviewed conversation shares the same suffix. Only sessions in which Auto actually reviewed something have such a lane.

Snapshot at 2026-11-04T05:45Z, covering 2026-10-22 → 2026-11-04:

Lane family	lanes	requests	input tokens	cache hit	input/request	spend (USD)	share
main conversations	162	5,155	883.5 M	97.9%	171k	5.978	57.0%
auto review	11	3,888	1,194.1 M	99.2%	307k	4.406	42.0%
proxy-side folding	1	20	7.6 M	93.3%	382k	0.106	1.0%

So the reviewer issues ~0.75 model requests per main request, each carrying more input than a main request. Spend is as the proxy reports it (derived from a price table, not from an invoice).

What one review request contains

One captured review request: 1,583,596 characters ≈ 790k tokens.

Section	share of payload
FILTERED_HISTORY	99.8%
PENDING_ACTION	0.2%
ENVIRONMENT / PROJECT_INSTRUCTIONS	≈ 0

Inside FILTERED_HISTORY:

Entry kind	characters	share of whole request
tool-call (arguments JSON)	1,301,342	82.2% (1,368 entries)
user-message facts	183,296	11.6%
human instructions	28,628	1.8%

The reviewer is therefore mostly re-reading the full arguments of every prior tool call — long shell scripts, file bodies, JSON payloads — not the conversation text.

Mechanism (from the shipped bundle)

• Hook: tools/pre-execute (dsh-experimental-auto-review/lib/index.js:462).
• Gate: :465 if (permissionPresets.current(agent.session) !== AUTO_PRESET) return next(); — only the Auto preset reviews.
• Granularity: once per supported call, including each started PTC inner call; identity is (turn, step, callId); there is no step-level batching.
• Cost note: README.zh.md:90 — "each supported call produces one additional model request; it is not cached, retried, truncated, or compressed, and it has no separate small output budget."
• Read-only is not exempt by rule: lib/index.js:39 records that "a production read is medium even though it is read-only".

On caching: the 99.2% hit rate means the repeated payload is cheap per token, but the volume still dominates — this is a request-count problem, not a pricing problem.

Why it seems worth a discussion

1. It is invisible in the bill. There is no per-feature line; the only way to find it is to attribute requests by prompt fingerprint or by token delta around tool calls.
2. It scales with tool-call count, not with conversation length — the more calls the agent makes, the larger this fixed tax.
3. 82% of what it pays for is tool-call arguments, which are plausibly the part of the context least likely to change an approval decision.

Reproducing

Any proxy that logs upstream requests can reproduce the volume: group requests by system-prompt hash within a session, then compare request counts against the session's own turn/step count. A cheaper check is to record the token delta per tool call under Auto versus any other preset.

Possible directions (not a demand)

1. Skip or batch review for calls whose risk is structurally known (read-only file reads, git status, listing).
2. Truncate tool-call arguments in the review snapshot — they are 82% of the payload, and head/tail plus a character count would usually suffice.
3. Offer a policy-file mode: static rules answer the common cases, and the model is consulted only when the rules are inconclusive.
4. Surface per-feature usage so users can see what the Auto preset costs before enabling it.

[dgesteves]

Nothing can exempt, batch or truncate reviews today: apply() takes no config and the README says so outright ("no deterministic tool exemptions, persistent grants, configurable policy"). What you can control is how big each review is, because FILTERED_HISTORY is built from the session's current surface (session.surface.nodes), not from the whole event log. So /compact shrinks every review after it.

I ran this on 0.2.0-rc.2 with Auto on, a local mock model behind DEEPSEEK_BASE_URL, and 100 KB tool arguments:

	FILTERED_HISTORY chars	tool-call entries
6th call in the session	503,791	5
next call after /compact	2,700	0

Without the /compact, a similar run on rc.5 kept climbing to 1,008,094 chars at 10 entries. With an rc.5 session on Workspace Write instead of Auto, three tool calls produced zero review requests, so switching with /permission for a read-heavy stretch removes the cost completely. These are character counts from a mock, so I can't speak to your real bill.

That also fits your 790k capture: automatic compaction defaults to thresholdRatio: 0.8 on a 1,000,000 window, so a long session's reviews can grow to roughly that size before anything trims them. Lowering the ratio alone doesn't reset it, though. With 0.2 on rc.5, compaction fired early but the next review still carried 6 tool calls, I assume because retainRatio (0.16) keeps recent context. The setting also lives inside the preset's plugins list, so overriding it means carrying a copy of the whole list.

One caution on truncating arguments (your direction 2, or doing it in your proxy): the policy classes "exact cleanup of an object that retained historical tool-call facts establish the agent created in this same session" as low risk. Those arguments are that evidence, so head/tail truncation could turn legitimate cleanups into denials. I haven't tested that; it's from reading the policy.

Sources at rc.5: surface snapshot, cleanup rule, known limitations, threshold default. The same snapshot line is in 0.2.0-rc.2.

[PedroCostinha]

Hi! The request attribution data is particularly useful because it separates the main model workload from the automatic reviewer.

I would suggest treating review overhead as a measurable feature budget.

For each review request, the runtime could record the originating tool call, input tokens, cache usage, estimated cost, and review decision.

A possible optimization would be a tiered policy:

1. Apply deterministic checks to operations whose risk can be classified safely.
2. Use model-based review for ambiguous or higher-risk operations.
3. Avoid sending unnecessary historical tool arguments when the current action can be evaluated with a smaller context.
4. Show review-related usage separately from main conversation usage.

Read-only operations should not be exempt automatically, since some can access sensitive data.

A useful benchmark would compare cost, latency, and authorization decisions before and after optimization using the same recorded test cases.

This would help reduce overhead without silently weakening the permission model.

[XuZhangxing]

这么夸张吗？0.o