dsh-token-meter: messageTokens can underflow below 0 (Zod nonnegative fails)

[myYangyunfan]

Symptom

Chat turns fail with a Zod validation error:

Too small: expected number to be >= 0

at path: ["messageTokens"], surfaced to the user as a failed turn.

Root cause

In packages/llm/token-meter, contextBreakdownProjectionDefinition.apply accumulates:

messageTokens: state.messageTokens + fold.deltaTokens,

foldSurfaceProjection returns a negative deltaTokens when a surface message is replaced/compacted (surfaceOp === "replace" → deltaTokens = tokens - claim.tokens). When the magnitude of that negative delta exceeds the currently accumulated state.messageTokens, the sum goes below zero and trips the tokenCount = z.number().int().nonnegative() schema on the next checkpoint/validation.

systemTokens / toolsTokens are always non-negative from their estimators; only messageTokens can underflow via the signed fold delta.

Suggested fix

Clamp the accumulator to a non-negative floor:

messageTokens: Math.max(0, state.messageTokens + fold.deltaTokens),

or make the shadow-price claim accounting resilient to a claim whose shadowedTokenCount exceeds the live accumulated total (the two should never disagree, but a mismatch currently hard-crashes the turn instead of degrading the heuristic figure).

Version

@deepseek-ai/dsh-token-meter@0.1.1-rc.2

Note: this figure is a heuristic context-composition estimate (display/metering only), so flooring it at 0 is safe and does not affect the actual request payload.

[weijiafu14]

你的根因和修法都对，而且隔壁 #4674 是同一个 bug、已经有了一份写好的补丁——先指过去省你时间，然后说一件你这份分析里需要修正的地方。

同一个 bug，那边已经走到参考实现了

#4674 报的是"大范围 compress 后 contextBreakdown.messageTokens 为负、zod 拒绝、会话卡死"，根因链和你写的一字不差（signed surface fold 的 deltaTokens = tokens - claim.tokens 为负、apply 里 state.messageTokens + fold.deltaTokens 无 clamp）。那边现在有：

• @nokkies 的独立复现：把上报的形状直接过一遍投影，得到 messageTokens = **-499982**，并指出状态 schema 和 wire view 都声明了 nonnegative，所以这不只是屏幕上一个错的数字——之后每一轮都过不了校验，会话永远继续不了；
• @Jstn-1g 基于 master b150a551b8 的有界参考实现（见下）。

你这份分析里有一处需要修正：不止 messageTokens 一个字段

你写的：

systemTokens / toolsTokens 都由估算器给出、总是非负；只有 messageTokens 会经由 signed fold delta 下溢。

前半对，但结论不完整。@Jstn-1g 的补丁覆盖的是共享那条 signed surface fold 的两个消费者：

1. contextBreakdown.messageTokens —— 你我都看到的那个；
2. contextPressure.surfaceTokens —— 同一条 fold 的兄弟消费者，它的 state schema 同样会拒绝负值。

也就是说：只 clamp 你这一个字段，另一条路上还会以同样的方式炸。 这一点值得你知道，因为按你现在的写法，一个照着改的人会以为一行 Math.max(0, …) 就完事了。

（他的补丁还多做了一件事：把两者的 checkpoint 版本从 2→3 和 4→5，好让旧的缓存行被丢弃、从 seq 0 重新折叠权威日志。这一步很关键——因为投影状态已经写进缓存了，光改代码不动版本号的话，已经处于负值状态的那些会话仍然是坏的。你原文和 #4674 都提到"会话卡死、只能放弃"，正是这个原因。）

你补的那句"这是启发式估算、只用于展示与计量"很重要

Note: this figure is a heuristic context-composition estimate (display/metering only), so flooring it at 0 is safe and does not affect the actual request payload.

这句话应该放在最前面，因为它直接回答了评审对 clamp 的唯一顾虑——"把一个错的数字掩盖掉会不会藏住真问题"。

答案是：这个数字本来就不参与请求组装，而现在的行为是让一个展示用的估算值把整轮运行搞失败。也就是说，当前设计把一个"显示可能不准"的问题升级成了"会话不可用"——这个定性比"数字会变负"有力得多，建议你和 #4674 那边都强调这一层。

（这属于这个社区里一个反复出现的形状：一个防御性校验发现了不一致，但除了"停"以外没有任何降级路径，于是它防住的东西比它造成的后果轻。 同族的还有 replay state 块数不匹配导致会话永久报废、图片会话切文本模型被拒导致无模型可用等好几例。你这条的特殊之处在于——它拒绝的是一个自己声明"仅供展示"的字段，所以 fail-soft 的正当性比其它几例都强。）

建议

把你这帖合并到 #4674 去（或者互相引用），并把上面两点带过去：

1. contextPressure.surfaceTokens 也要 clamp（如果 @Jstn-1g 的补丁没被采纳，这一点会丢）；
2. "仅供展示/计量"这句定性——它是支持 fail-soft 的最强论据，而 dsh: 大范围 compress 后 contextBreakdown.messageTokens 为负，zod 拒绝导致本轮运行失败（会话卡死） #4674 那边目前没有人明确说出来。

边界与利益相关

我们不修 DSH 自家组件——token-meter 的投影与 schema 都在 DSH 里。上面全是转述 #4674 的进展加两条对照；我没有复现过这个下溢，也没有跑过 @Jstn-1g 的补丁。

利益相关：我维护 pi2dsh（Pi 生态兼容层）。这条不推销——这是 DSH 自家 token meter 的投影问题，装什么插件都不改变那个累加器。

[denial123789]

The fail-soft direction is appropriate because this is heuristic projection state, but the complete recovery boundary spans more than one accumulator. The shared signed fold also feeds contextPressure.surfaceTokens, so guarding only contextBreakdown.messageTokens leaves a sibling nonnegative schema exposed. Existing Sessions also need projection checkpoint version migration; otherwise reducer code changes do not prove cached rows will be discarded and refolded from authoritative history.

A bounded recovery test should preserve a copied Session, clamp both consumers, advance their independent checkpoint versions, refold from seq 0, parse both state/wire views, and complete a subsequent model turn. We updated the source-pinned accounting guide with containment and these recovery gates: https://github.com/sandbaseai/deepseek-harness-handbook/blob/main/docs/en/operations/token-meter-accounting.md

Disclosure: I contribute to this independent community handbook.

[nokkies]

Confirmed, and your root cause is exact — messageTokens is the only one of the three that can go negative, because it is the only one fed by a signed fold delta rather than an estimator.

This is the same defect as #4674, which has more of the failure shape recorded if it is useful: the reason it is unrecoverable rather than merely wrong is that the clamp is missing on BOTH sides of the wire. The state schema and the view schema each declare nonnegative, so once the accumulator goes below zero, zod rejects it on every following turn — the conversation cannot continue at all, and there is no way back short of editing the log.

Worth adding to your analysis: the overshoot is not necessarily a bug in the delta. A replacement's delta is summary - claim, where the claim prices everything the compaction shadowed, while the running total holds only the appends that projection folded. Those are not the same set. A checkpoint resumed mid-session, or a compaction reaching back over history the current projection never folded, makes the claim legitimately larger than the total — so a correct delta can still drive a correct accumulator negative. That is why clamping is the right fix rather than hunting for the wrong subtraction:

messageTokens: Math.max(0, state.messageTokens + fold.deltaTokens),

Under-reporting until the total recovers is strictly better than a permanently unusable session, for a number that is an estimate to begin with.

Fixed locally exactly that way. The reproduction that pins it is a compaction whose claim exceeds the folded total — not a large session, which is what makes it look rare when it is really just conditional.