[Bug] pi-ai 用估算值把 max_tokens 填满窗口（固定 4096 余量），导致确定性超窗 400

[yang-tsao]

摘要

@earendil-works/pi-ai@1.1.0（DSH 经 @deepseek-ai/dsh-llm-pi-ai 使用）在 clampMaxTokensToContext 中用本地启发式估算把 max_tokens 填满整个上下文窗口，只保留固定的 CONTEXT_SAFETY_TOKENS = 4096。只要估算比服务端真实计数低超过 4096，拼出的请求就是 输入 + 输出 = contextWindow + 1，被 provider 以 400 拒绝。这是确定性的客户端缺陷，与具体 upstream 无关。

环境

项	值
DSH	桌面端 0.2.0-rc.2（desktop profile）
适配器	@deepseek-ai/dsh-llm-pi-ai → @earendil-works/pi-ai@1.1.0
路由	openrouter / z-ai/glm-5.3-flash
profile 声明容量	contextWindow: 1048576，maxTokens: 943718
上游	OpenRouter 网关 → Parasail（其中一次报错由网关自身返回）

现有 patches/@earendil-works__pi-ai@1.1.0.patch 未触及该函数（它只删除流式 tool 参数的逐 delta JSON 解析）。

现象

同一 session 内两次 400，算术完全一致（W = 1048576）：

	第一次	第二次
provider 报告的 output 请求	742240	786743
反推 pi-ai 估算 est = W − 4096 − output	302240	257737
est + 4096（客户端认为的总量）	306336	261833
服务端实际计入的输入	309554	261834
超出窗口	+3218	+1

第一次的 provider 报文自己给出了构成：1051794 tokens (193907 text input, 115647 tool input, 742240 in the output) —— 被低估的正是 tool 输出。

第二次原文：

This model's maximum context length is 1048576 tokens. However, you requested 786743 output tokens
and your prompt contains at least 261834 input tokens, for a total of at least 1048577 tokens.
(parameter=input_tokens, value=261834)

外层由 OpenRouter 包装：{"message":"Provider returned error","code":400,"metadata":{"raw":"…","provider_name":"Parasail","is_byok":false}}。

根因

随包发布的 pi-ai（dist/utils/simple-options.js / dist/utils/estimate.js）：

const CONTEXT_SAFETY_TOKENS = 4096;
const MIN_MAX_TOKENS = 1;

function clampMaxTokensToContext(model, context, maxTokens) {
    if (model.contextWindow <= 0)
        return Math.max(MIN_MAX_TOKENS, maxTokens);
    const available = model.contextWindow - estimateContextTokens(context).tokens - CONTEXT_SAFETY_TOKENS;
    return Math.min(maxTokens, Math.max(MIN_MAX_TOKENS, available));
}

调用点 buildBaseOptions：

maxTokens: clampMaxTokensToContext(model, context, options?.maxTokens ?? model.maxTokens),

estimateContextTokens 优先复用最近一条 assistant 的 usage，再对其后的消息做启发式估算（estimateMessageTokens / estimateTextTokens）；tool 结果恰好落在被估算的那一段。因此当估算偏差 ε > 4096 时，请求总量必然是 W + (ε − 4096)。

DSH 侧没有本地兜底：仓库内 grep clampMaxTokens / CONTEXT_SAFETY_TOKENS 无结果。

安全网为什么没兜住

该 session（900 个 step）共 136 次 compaction/start，只有 99 次干净收尾：Stream ended without finish_reason ×8、pi-ai stream idle timeout ×2、terminated ×2、summary is not smaller than the shadowed content ×6、summarization truncated at the token cap ×2。主路由 assistant/attempt 失败中 TRANSPORT ×28、TIMEOUT ×4。

失败前后的事件序列（CONTEXT_WINDOW_EXCEEDED 已被正确归一化，溢出恢复也确实启动了）：

6694 compaction/start
6696 compaction/end    error: summarization truncated at the token cap (incomplete checkpoint)
6698 assistant/attempt → 400 CONTEXT_WINDOW_EXCEEDED
6699 compaction/start  (overflow recovery)
6700 compaction/end    error: Stream ended without finish_reason
6702 turn/end          error

恢复用的 summarizer 自身流中断 → 没有持久缩减 → 不授权 retry → 原始 400 抛给用户。

放大因素

profile 给该模型声明 maxTokens: 943718（窗口的 90%）。@deepseek-ai/dsh-compaction-basic 的阈值是

thresholdTokens = floor(min(W × thresholdRatio, W − O − headroomTokens))
                = min(838860, 1048576 − 943718 − 65536) = 39322

即 harness 认为 prompt 必须压到约 39k，而模型实际能接受接近 1M 的输入。结果是每一步都在压缩（turn 62 有 66 个 step）；压缩一旦失败，请求就带着约 300k 的 prompt 出去。此处 O 直接取自请求 envelope 的 maxTokens，所以把 maxTokens 声明为窗口的 90% 会把压力预算压到几乎为零。

建议

1. pi-ai：余量不要用固定 4096，按估算比例缩放（如 max(4096, 2% × estimate)）；或不要“填满窗口”，保留固定比例。
2. DSH：dsh-llm-pi-ai 在传入 options.maxTokens 前预留安全比例；现有 patch 文件是合适的落点。
3. 文档/校验：模型目录若把 maxTokens 生成成窗口的 90%，会让上述阈值公式塌陷，建议给出上限建议或告警。

复现要点

任意 pi-ai 路由 + tool 输出占比高的 prompt，使本地估算低估超过 4096 tokens 即可命中。本案例中 objdump 反汇编文本被大量回灌，估算偏低 4097（另一处 7314）。

[argszero]

Every source claim in this report reproduces on master @ 571902028a75c64c1d767f4ae5fd099485f0ddf0, including the estimator at the centre of it. Two of your numbers I can sharpen, and one half of the chain turns out to be doing the right thing already — so I would rather name that than leave you expecting a fix where none is needed.

The mechanism, verified from the published tarball

dist/api/simple-options.js (not dist/utils/ — only the estimator is there) is exactly as you quote it, and dist/utils/estimate.js is what makes the error term systematic rather than random:

const CHARS_PER_TOKEN = 3.5
export function estimateContextTokens(context) {
  const usageInfo = getLastAssistantUsageInfo(messages)   // newest assistant message that is
  if (usageInfo) {                                        // neither "aborted" nor "error",
    …                                                     // with a non-zero usage total
    return { tokens: usageTokens + trailingTokens, … }
  }
  …                                                       // otherwise: pure character estimate
}

getLastAssistantUsageInfo reuses the provider's own usage for the newest assistant message whose stopReason is neither aborted nor error, then prices everything after it at ceil(chars / 3.5). Your tool output sits entirely in that trailing span, and a disassembly listing is far denser than one token per 3.5 characters — which is the whole of your "the underestimated part is precisely the tool output" (translated from the Chinese original). So ε is not a rounding wobble: it scales with how much of the trailing span is dense, machine-generated text, and it can only ever be an undercount for that kind of content.

Your own arithmetic then follows exactly: available = W − est − 4096, and clampMaxTokensToContext returned 742,240 / 786,743, so it took the available branch (min(maxTokens, available)), giving est = 302,240 / 257,737, against the provider's 309,554 / 261,834 — an undercount of 7,314 and 4,097, i.e. 2.42% and 1.59% of the estimate.

That last pair is worth having before your suggestion 1 is written: max(4096, 2% × estimate) would have left 6,045 of margin against a 7,314 error, so the first refusal still happens under that rule. A proportional margin is the right shape — the error is proportional to the estimated span, not constant — but 2% is under both of your own measurements' larger term; the stricter of the two is what the rule has to clear.

The patch, and where a fix can actually land

The patch does not touch the clamp — confirmed, with one file more than you listed. patches/@earendil-works__pi-ai@1.1.0.patch touches six files (anthropic-messages.js, bedrock-converse-stream.js, mistral-conversations.js, openai-completions.js, openai-responses-shared.js, pi-messages.js), one line each, all removing per-delta tool-argument JSON parsing; a search for clamp / estimate / simple-options / CONTEXT_SAFETY over it returns nothing.

Where a fix lands matters as much as whether one exists: patchedDependencies is a pnpm-workspace setting (pnpm-workspace.yaml:108-109), so the patch applies where this repository installs — the desktop build included — and does not travel to anyone who installs the published packages, whose @earendil-works/pi-ai is the pristine one. A repo patch and a package-level fix are two different deliverables, and only the second reaches npm consumers. Your suggestion 2 is well-placed for that reason.

The compaction threshold: verbatim, and closer to a hard error than the report says

packages/compaction/compaction-basic/src/config.ts:191-194 is your arithmetic, character for character:

const thresholdTokens = Math.floor(Math.min(
  contextWindow * policy.thresholdRatio,
  pressureBudgetTokens,
))

with messageBudgetTokens = contextWindow − reservedCompletionTokens (:172) and pressureBudgetTokens = messageBudgetTokens − policy.headroomTokens (:181). The reservation's own docstring (src/index.ts:59-64) states your reading in the same words: "The effective envelope's own cap wins; otherwise the adapter's per-request default, which the adapter materializes when that envelope omits one. No declared cap means no reservation." It is called with info.defaultMaxTokens at :317, so O really is the envelope's maxTokens when one is set, and min(838,860, 39,322) = 39,322 is right.

What the report does not say is that this function throws on both ends, and the reported catalog is one step from one of them:

• config.ts:182-189 — O ≥ W − headroomTokens leaves no pressure budget: "reduce the effective request maxTokens or compaction headroom tokens, or configure a larger adapter model contextWindow". For your window that cliff is 983,040, i.e. 93.75% of the window; a catalog at 90% (943,718) sits 39,322 tokens below it — the same small number your min() selects, which is not a coincidence: at thresholdRatio 0.8 the pressure budget and the distance to the cliff are one expression. Any catalog that emits more than 93.75% of any window fails at resolve time on this path.
• config.ts:172-180 — O ≥ W is the earlier cliff ("leaving no message budget").

So "give the catalog a documented upper bound" is not tuning advice: it is the difference between a session that compacts constantly and one that does not start.

The classification half is already correct — and the gap is next to it

Your event sequence shows the 400 arriving as CONTEXT_WINDOW_EXCEEDED and the overflow recovery starting, and compaction-basic's agent/request-error handler gates on exactly that code (src/index.ts:194). So this is not a classification gap, and I would rather say so than let you install something expecting otherwise: the community packages in that family exist for the opposite case — a provider refusing for "too long" reasons while the harness files it under some other code. Here the code was already right.

What your log shows is the other gap in that same handler. Read src/index.ts:206-227: when compactIfNeeded throws and session.surface.replaceGeneration has not advanced, it logs "preserving the original request error" and calls next() — the retry is declined because nothing durable shrank. That is exactly your state:

6696 compaction/end  error: summarization truncated at the token cap
6700 compaction/end  error: Stream ended without finish_reason

The summarizer's own stream died, no durable reduction landed, so the shipped recovery yields and the raw 400 reaches you — "no durable reduction ⇒ no retry authorized ⇒ the original 400 is thrown to the user" (translated from the Chinese original). That fall-through is the trigger condition of @argszero/dsh-overflow-recovery (repo): mount it and it runs after the shipped handler (next() first, then decide — so listener order does not matter), and when the chain produced no decision it commits a deterministic, model-free reduction of the model-visible surface. No summarizer call at all, so a summarizer that cannot hold a 300k prompt cannot block it. It is a fallback rather than a fix, and it is lossy in a way it names in the transcript, but it is aimed precisely at the line above.

Prevention: the one request-side lever a plugin owns

Suggestion 1 is upstream's and I agree with it. For the local half, the seam is newer than most reports assume:

prepareCall(config: LlmCallConfig, signal?: AbortSignal, configure?: ConfigureCall): Promise<PreparedLlmCall>

(packages/llm/llm/src/index.ts:956-960), reached by the loop at packages/core/agent-loop/src/agent.ts:604, where

type ConfigureCall = (controls: Readonly<LlmCallControls>, model: Readonly<LlmResolvedModelInfo>) => LlmCallControls

and LlmCallControls carries maxTokens (llm/src/call-config.ts:16-21, :45-48). It runs before defaults and validation, and "later writes replace earlier ones" (:38). A plugin that shadows the runtime instance can compose a configure of its own and lower the outgoing cap — reserving room before the request is assembled, instead of classifying its rejection afterwards. agent/request replaces the harness envelope, not the adapter's derived cap, so this is the only request-side lever I have found that a plugin owns.

Its honest limit is your own defect: such a plugin's budget comes from the harness's token meter, which is an estimate, and your error term is an estimate that undercounted dense tool output by 2.42% and 1.59%. A clamp driven by the same class of number needs a conservative margin and should be observable (report before enforce), which is why I would describe the seam here rather than claim it solved.

One more fact from the adapter's own README (packages/llm/llm-pi-ai/README.md:105) that belongs beside the analysis: "Reasoning tokens at any selected level count toward the request's output cap." With reasoningEffort: max/xhigh, part of that 742,240-token output request is thinking, so the reasoning-volume block in your log and the reserve error are two readings of one budget.