Fix token overcounting from streaming duplicate JSONL entries#278
Conversation
Claude Code writes multiple JSONL entries per API call during streaming: intermediate entries have stop_reason: null, and only the final entry has a string value like "end_turn" or "tool_use". The getTokenMetrics function was summing all entries, inflating the total by ~2.5x. Now only counts final entries (those with a truthy stop_reason). Falls back to counting all entries when no stop_reason data is present, for backward compatibility with older transcript formats. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4c0eaf6ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| for (const data of parsedEntries) { | ||
| // Skip intermediate streaming entries when stop_reason data is available | ||
| if (hasAnyStopReason && !data.message!.stop_reason) { |
There was a problem hiding this comment.
When any entry has a truthy stop_reason, the new filter skips all entries where stop_reason is falsy (if (hasAnyStopReason && !data.message!.stop_reason)), which also drops legacy entries where the field is absent. In a mixed-format transcript (e.g., older lines without stop_reason plus newer streaming lines), those legacy usage rows are excluded from totals and contextLength, causing undercounting despite the backward-compatibility goal.
Useful? React with 👍 / 👎.
|
Thanks for this, it will be published in the next release |
- fix: Fix token overcounting from streaming duplicate JSONL entries (sirmalloc#278) - fix: strip parenthetical suffix from model display name (sirmalloc#283) - Version bump, README cleanup Fork adaptation: Model.ts preserves [1m] suffix while adopting upstream's general parenthetical-strip regex. Model.test.ts updated to match fork behavior (ResetTimerWidget depends on [1m] for charged-model detection).
2 participants
Summary
stop_reason: null, final entries have"end_turn"or"tool_use")getTokenMetricswas summing all entries, inflating thetokens-totalwidget by ~2.5xstop_reason, with fallback to counting all entries when nostop_reasondata is present (backward compatibility with older transcripts)Before/After (real session, 21 API calls)
Changes
src/types/TokenMetrics.ts— Addedstop_reasontoTranscriptLine.messagesrc/utils/jsonl-metrics.ts— Two-pass approach: first scan detects if any entry hasstop_reason, then second pass skips intermediate streaming entriessrc/utils/__tests__/jsonl-metrics.test.ts— AddedstopReasonparam to test helper, two new tests (streaming dedup + legacy fallback)Test plan
stop_reasonfall back to counting all entriesjsonl-metrics.test.tstests pass (23/23)🤖 Generated with Claude Code