[FEATURE] JSONL usage.input_tokens are placeholder values; need complete token accounting

[Magnus-Gille]

Preflight Checklist

• I have searched existing requests and this feature hasn't been requested yet
• This is a single feature request (not multiple features)

Problem Statement

Claude Code's JSONL conversation logs record usage.input_tokens as a streaming placeholder (typically 1) rather than the final value. The remaining output gap (10-17x) is consistent with thinking tokens being excluded from usage.output_tokens. Together, this makes JSONL unreliable for token accounting.

I discovered this while building an energy monitor that reads the statusbar's context_window totals. When I independently summed JSONL tokens (deduplicated by requestId), the results diverged significantly from statusbar cumulative totals — but in a revealing pattern:

Feb 20 (heavy Opus 4.6 day, 20 sessions, 1,365 unique requests):

Metric	JSONL (dedup)	Statusbar	Ratio
Input tokens	41,444	7,199,162	174x
Output tokens	183,829	3,208,365	17x
Cache read	104,353,324	114,798,863	1.1x
Cache creation	3,170,696	2,717,775	0.9x

Feb 24 (12 sessions, 1,228 unique requests):

Metric	JSONL (dedup)	Statusbar	Ratio
Input tokens	11,758	1,193,366	102x
Output tokens	69,449	748,337	11x
Cache read	74,254,777	67,710,877	0.9x
Cache creation	2,817,739	2,003,545	0.7x

Cache metrics match closely (~1x), confirming both sources track the same API calls. But input tokens diverge by 100-174x, and output by 10-17x.

Root cause: 75% of JSONL entries have usage.input_tokens of 1 or 0 — these are streaming placeholder values that never get updated to the final count. JSONL also logs streaming duplicates (same requestId appears 2-10 times with identical placeholder values).

Proposed Solution

Two improvements would make JSONL useful for token accounting:

1. Write final usage values. After a request completes, update the JSONL entry (or append a final entry) with the real input_tokens and output_tokens including thinking tokens.

2. Deduplicate streaming entries. Either log only the final state per request, or add a "final": true marker so consumers know which entry to use.

Alternatively, a separate append-only usage log would work:

{"ts":1740000000,"requestId":"req_...","model":"opus","input_tokens":1234,"output_tokens":567,"cache_read":50000,"cache_creation":800}

Alternative Solutions

I read the statusbar's context_window.total_input_tokens / total_output_tokens via a custom statusline script, which gives accurate cumulative totals. But this only provides session-level aggregates, not per-call breakdowns.

I also verified the discrepancy using an independent JSONL parser (no third-party dependencies), reproducing the same pattern across two different days.

Priority

Medium - Would be very helpful

Feature Category

API and model interactions

Use Case Example

1. I use Claude Code daily and want to track my compute footprint
2. I sum token usage from JSONL logs — the only per-call data source available
3. The result is 100-174x too low on input tokens because usage.input_tokens is a placeholder
4. Tools like ccusage that read JSONL are affected by the same issue
5. With accurate final values in JSONL, any tool could provide reliable usage tracking

Additional Context

• Independent JSONL parser with deduplication: sum_jsonl.py
• Statusbar token semantics validated via analyze_tokens.py: total_input_tokens = fresh input only (excludes cache), total_output_tokens includes thinking
• Full investigation: FINDINGS.md
• Cache metrics matching at ~1x across both data sources confirms they're tracking the same API calls — the discrepancy is in the recorded values, not missing entries
• The output gap (10-17x) is partially explained by thinking tokens being included in statusbar but not JSONL, consistent with ~60-70% thinking overhead on Opus

[github-actions]

Found 3 possible duplicate issues:

1. [BUG] Streaming output tokens are incomplete in session.jsonl compared to /cost #21971
2. [BUG] JSONL session transcripts missing final message_stop event, output token counts unreliable #27361
3. JSONL session logs record placeholder output_tokens (always 1-2) instead of real values #25941

This issue will be automatically closed as a duplicate in 3 days.

• If your issue is a duplicate, please close it and 👍 the existing issue instead
• To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

[Magnus-Gille]

Not a duplicate. The linked issues (#21971, #27361, #25941) all report the output token streaming finalization bug
with a ~2x undercount. This issue documents two additional findings that none of them cover:

1. input_tokens is a placeholder (100-174x undercount). 75% of JSONL entries have usage.input_tokens of 0 or 1 — a
   streaming placeholder that is never updated to the final value. None of the linked issues mention input tokens;
   [BUG] JSONL session transcripts missing final message_stop event, output token counts unreliable #27361 explicitly states "input/cache token counts are accurate."
2. The output gap is 10-17x on thinking models, not ~2x. The linked issues predate Opus-class thinking models. On
   Opus 4.6, thinking tokens are 60-70% of output and are excluded from JSONL's output_tokens, widening the gap far
   beyond the ~2x streaming bug alone.

Cache metrics (cache_read_input_tokens, cache_creation_input_tokens) match ~1x between JSONL and statusbar,
confirming both sources track the same API calls. The problem is specifically that input_tokens and output_tokens
are never finalized.

This has practical impact beyond cost tracking — downstream analyses like Couch (2026) on Claude Code energy
consumption rely on JSONL token data and are undercounted by an estimated 4-6x as a result.

[github-actions]

Closing for now — inactive for too long. Please open a new issue if this is still relevant.

[github-actions]

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.