tokenMeter under-prices CJK text by ~2.3x: the chars/4 heuristic is script-blind

[SparkElf]

Summary

tokenMeter's fixed CHARS_PER_TOKEN = 4 heuristic systematically under-prices CJK content. Measured against a live OpenAI-compatible route (deepseek-v4.1-flash), the same request over-counts English text by ~1.4x and under-counts Chinese text by ~2.3x. On a real agent session whose surface was 5.3% CJK, tokenMeter estimated 106,241 tokens while the upstream billed 137,416 — a 22.7% shortfall.

This matters because compaction-basic compares measurement.totalTokens against thresholdTokens to decide whether to compact, but sizes the compaction span from the same estimate. When the estimate is low, a session sits below the trigger while the route sits above it, and each compaction removes a span priced by the low estimate.

Measured evidence

Density probe against POST /v1/responses (identical request shape, only the text differs; baseline = a single character):

Content	Characters	Billed prompt_tokens	chars/4 would say	Ratio
baseline (x)	1	5	—	—
Chinese sentence x40	1,120	644	280	2.3x under
English sentence x27	1,971	356	493	0.7x over

Per-character rates: Chinese 0.57 tokens/char, English 0.18 tokens/char. CHARS_PER_TOKEN = 4 (0.25 tokens/char) sits between them, so it is neither a conservative nor an accurate estimate — it is wrong in opposite directions depending on content.

Full-session cross-check (@deepseek-ai/dsh-token-meter 0.2.1-alpha.2, 430 surface nodes):

tokenMeter estimate : 106,241
upstream usage      : 137,416
shortfall           :  31,175 (22.7%)

The session's surface was 5.3% CJK / 93.8% ASCII. A naive content-weighted prediction from the probe rates gives ~25% shortfall, matching the observed 22.7%.

Why it is hard to notice

The shortfall is proportional to the share of CJK text, so it is invisible in English-only sessions and grows as a conversation turns Chinese. It also compounds with any content the estimate does not price at all:

• estimateContent walks message.content blocks only. Fields that still reach the request (for example the adapter's developer/system message assembled from the session, tool schema framing) are priced by estimateToolsTokens and estimateSystemMessage separately, which is correct — but the per-message container overhead beyond BLOCK_OVERHEAD is not.
• Embedded images are priced by estimateStructuralBlock at reference cost, which is right for a route that prices images as images. On a route that cannot read the Responses API array form of function_call_output.output, the route bills the base64 as text instead; that is a route-capability bug, not a meter bug, but the meter cannot see it and the compaction loop then works from a value that is orders of magnitude too small.

Reproduction

1. Any route where Chinese text is expected (a Chinese-language agent session is enough).
2. Log ctx.tokenMeter.measure(session).totalTokens and the usage.inputTokens of the next assistant/message.
3. Compare: the gap tracks CJK share.

Or use the density probe directly — send the two 2 KB strings above to /v1/responses and compare usage.prompt_tokens against ceil(len(text)/4).

Suggested direction

CHARS_PER_TOKEN is a single global constant, but the correct density is a property of the script mix, not of the model. Something like an ASCII-vs-CJK aware estimate (for example asciiChars/4 + cjkChars/1.75, calibrated from the probe above) would remove the systematic error without needing a real tokenizer. If a real tokenizer is out of scope, a per-route density multiplier would at least let a deployment correct for its own traffic.

Alternatives considered and rejected: keeping chars/4 and raising headroomTokens only makes the slop larger; compacting earlier is a global tax on every session to cover a defect that only affects CJK ones.

Environment

• @deepseek-ai/dsh-token-meter 0.2.1-alpha.2
• @deepseek-ai/dsh-compaction-basic 0.2.1-alpha.2
• Route: OpenAI-compatible openai-responses, contextWindow: 640000, maxTokens: 32000, model deepseek-v4.1-flash
• Cross-checked against a second independent gateway for the same request: identical prompt_tokens, so the count is the model's, not the gateway's

(Related: #9041 covers a threshold collapse from a large route maxTokens; #8088 reports early compaction at ~50% context; #4454 covers the summary is not smaller abort. This is a different defect — the estimate itself, not the threshold arithmetic.)

[PerryLink]

The premise is confirmed — CHARS_PER_TOKEN = 4 is real and script-blind — but there are two things about the location that change how the report should be filed.

Where it lives

packages/llm/token-meter/src/estimate.ts:13:

const CHARS_PER_TOKEN = 4
const BLOCK_OVERHEAD = 4          // :16
export const ROLE_OVERHEAD = 4    // :19

and the constant is applied uniformly — text blocks (:48), tool names and arguments (:51-52), JSON-stringified reference and generic blocks (:32, :34), and the tool header (:101). So there is no per-script path anywhere in the estimator, which is exactly your point. Your 0.57 tokens/char measurement for Chinese against 0.25 implied is a 2.28x under-count, so the "~2.3x" in your title is right.

There are two estimators, and they disagree

In the same tree, the pi-ai adapter's estimator uses a different constant: CHARS_PER_TOKEN = 3.5 (@earendil-works/pi-ai, dist/utils/estimate.js:2), and that one feeds clampMaxTokensToContext. So the harness prices context at 4 chars/token while the adapter sizes its max-token clamp at 3.5 — a ~12.5% disagreement on pure ASCII, before any CJK effect is considered. Both are script-blind, in the same direction, with different magnitudes.

That is worth a line in your report because it means a fix in one place leaves the other: correcting token-meter does not change what pi-ai clamps, and correcting the clamp does not change what compaction decides.

On the impact claim

Your chain — estimated totalTokens below the real billed count, compaction threshold not reached, compaction span sized from the same low estimate — is the right one, and compaction-basic does compare measurement.totalTokens against thresholdTokens. But two cautions:

• The compaction span is not only an estimate-driven quantity; it is also bounded by the retained-tail policy and by requests actually being persisted in the session. A low estimate shifts when compaction triggers, not necessarily by a proportional amount what it removes. Worth separating those two effects, because a maintainer reading "each compaction removes a span priced by the low estimate" will ask for the second half of that argument.
• Your measured "22.7% shortfall on a 5.3% CJK surface" is a much stronger and more portable number than the per-character table. If the report leads with the session-level comparison and uses the density probe only as the explanation, it is harder to dismiss as a synthetic measurement.

A framing note

"Under-prices" implies the cost is the problem; the durable problem is that compaction decisions are made against a number that is wrong by a factor that depends on the user's language. A user writing Chinese gets a systematically different compaction policy from a user writing English, on the same session size. That is the sentence I would put at the top.

[IlyaKhalafi]

The pure-CJK probe shows a weakness in chars/4, but I would separate that from the full-session comparison before choosing a correction factor.

I checked the published 0.2.1-alpha.2 meter. It already uses provider usage as a baseline when the request envelope matches the last successful call and that usage is at least the heuristic price of the anchor. The same logic is visible here. totalTokens includes that baseline plus the signed surface delta. The prior output belongs in that baseline because it becomes input to the next request.

Log baseline, surfaceDeltaTokens, logRevision and the request envelope just before sending. Compare totalTokens with the next call's normalized inputTokens + cacheReadTokens + cacheWriteTokens, treating absent cache buckets as zero. Do not add cache again to a raw provider total that already includes it. This separates heuristic fallback from growth after a usage anchor.

There is also a mismatch in the mixture calculation. If your percentages are character shares, 0.053 * 0.57 + 0.938 * 0.18 gives about 0.199 tokens per character, below 0.25. Those known 99.1% alone do not predict a 25% shortfall. The remaining characters and request serialization still need accounting for before attributing the session's gap to CJK density.

[PerryLink]

Both points hold, and I checked them against the code rather than just against the published package.

The anchor gate is packages/llm/token-meter/src/index.ts:158: the usage baseline is taken only when an anchor exists and optionalHeaderEquals(anchor.header, header). Inside that branch the heuristic prices the anchor surface plus the routed tool header (:162-164), the baseline becomes usageTokens(usage) when it is at least that anchor price (:168-170), and the only heuristic quantity that survives is the signed surface delta (:171). totalTokens is baseline plus that delta (:187).

So the constant cannot produce a whole-session shortfall on its own. A 22.7% gap between totalTokens and the next call's billed input has to come from a measurement where the anchor branch did not run: no anchor yet, a header that does not compare equal, or content the surface pricing cannot see at all. Your two-line diagnostic separates exactly those, and usageTokens at :71-76 is the right normalisation for the right-hand side, since it is input plus cache-read plus cache-write plus output.

On the arithmetic you are also right, and this one is my error rather than the reporter's. 0.053 * 0.57 + 0.938 * 0.18 is about 0.199 tokens per character, which is below the 0.25 that chars/4 implies. A mixture that is 5% Chinese should therefore make the heuristic over-estimate slightly, not under-estimate by a quarter. I quoted the report's "naive content-weighted prediction gives ~25% shortfall" line back at it without doing the multiplication, and the sentence is sign-flipped; it is worth correcting in the thread so nobody cites the 22.7% later as a measured CJK effect.

What survives is narrower than the title suggests but still real: the constant is script-blind, the block-level 2.28x under-count for pure CJK stands, and the exposure is the cold or full-reprice measurement plus one step's growth. One detail that cuts in your favour and is worth stating: because the usage baseline is only adopted when usage is at least the full heuristic anchor price (:168), a heuristic that runs low biases toward the usage path rather than away from it. Under-counting makes the anchor win more often, which is the opposite of the failure mode the report describes.