[token-meter] Stale usage anchor after compaction surface-replace → large over-estimate on non-Latin content; baseline.kind keeps saying 'usage'

[iia-arg]

Filed as a Discussion because Issues are disabled for this repository — CONTRIBUTING.md:11 and
README.md:45 point bug reports here.

Versions: observed on @deepseek-ai/dsh-token-meter 0.1.1-rc.2 with
@deepseek-ai/dsh-compaction-basic 0.1.1-rc.2. The same anchor / surface-replace logic is present
in the currently published 0.1.2-rc.1 and in master (0.1.3-alpha.1, commit 14befe7). In 0.1.2+ the
kind is recomputed rather than copied, but from the same stale anchor usage and the same
pre-compaction surface snapshot — the form changed, the outcome did not.

What happens

measure() computes totalTokens = anchor + (surfaceTokens - surfaceTokens_at_anchor),
where anchor is the provider usage of the last assistant/message. compaction/summary
is applied as a surface-replace: removed nodes are subtracted at their HEURISTIC price
(CHARS_PER_TOKEN = 4), the summary is added at its own heuristic price. The anchor itself
is only updated on assistant/message — never on compaction/summary.

So compaction IS accounted for, but the removed content is subtracted at the wrong price.

Observed numbers (one real compaction, mixed Cyrillic + JSON payload)

anchor before compaction (provider usage):        805 004
shadowedTokenCount reported by summary:            41 253   <- heuristic, 58 nodes
measure() right after compaction:                 775 468
first provider usage AFTER compaction:            433 830   <- real window

The provider says roughly 383 000 tokens were removed. The heuristic estimated 41 253 —
about 9x low on this content. CHARS_PER_TOKEN = 4 is far from the real tokenizer ratio for
Cyrillic text and JSON.

The window of the wrong value is exactly one step: it closes as soon as the next
assistant/message lands and re-anchors. Four consecutive calls within that step returned
the identical number, which is what made it visible — a live window does not stand still.

Second consequence: the usage label outlives the event that invalidates it

baseline is copied from the anchor wholesale, kind included (baseline = anchor.baseline),
and the anchor is only ever set on assistant/message. So after a surface-replace the meter keeps
reporting baseline.kind === 'usage' — i.e. it still claims provider-grade numbers — while the
underlying usage was taken before the replace.

Any consumer that shows or trusts baseline.kind is told the measurement is provider-backed at
exactly the moment it is not. This is worse than a missing freshness flag: the flag is present and
actively reassuring. Observed directly — four consecutive calls in that one step printed the
provider-backed label next to the stale number.

Why it is not cosmetic

dsh-compaction-basic calls the same measure() right after compacting, and if the value is
still >= threshold it compacts AGAIN (retry, default 1), then throws "still above threshold".
With an over-estimate of this size, a run that legitimately dropped well below the threshold
can be measured as still above it and be compacted a second time — discarding the fresh summary,
spending extra model calls, and losing context. In the observed case it passed with a margin of
24 532 tokens. That margin is luck, not headroom: because measure() tracks the anchor
one-for-one, an anchor just 24 532 higher would have put the post-compaction measurement exactly
at the threshold and fired a second pass.

Related discussions

• [token-meter] Char-heuristic estimation diverges substantially from provider-reported usage tokens #3514 — char-heuristic vs provider usage. Anchoring on usage landed in rc.8; what follows
  is the case inside that path: the anchor is taken, but after a surface-replace it is neither
  reset nor re-priced, and the usage label survives with it.
• Six implementation findings with fix proposals (0.1.1-rc.2, Windows) #5198, item 2 — CHARS_PER_TOKEN = 4 underestimates CJK by roughly 4x. The 9x measured here
  on Cyrillic + JSON is the same class on a different corpus, and supports the weighting patch
  proposed there.
• `tokenUsage` projection never folds `compaction/summary.usage`, and replaces rather than adds across a retried step #1886 — compaction/summary.usage exists but is never folded into the projection. Fix (a)
  below builds directly on that: the data needed to re-anchor is already written.
• [严重 Bug] 中文/工具密集长会话出现 compaction thrashing：持续阻塞正常使用 #2737 — compaction thrashing reported in the field (provider 239 617 vs heuristic 47 187).
  Same gap, different symptom: there it re-compacts every step, here it can double-compact once.

Suggested fix (either is enough)

• Re-anchor on the summary call's own usage when the provider reports it for the
  compaction request, or
• Mark the baseline as estimated after a surface-replace and have consumers (including the
  post-compaction threshold check in dsh-compaction-basic) refuse to compare against the
  threshold until the first real usage arrives.

The second is safer: it makes the unknown explicit instead of replacing one estimate with another.