Codex subagent fork history causes massive Cache Read overcount

[pitimon]

Summary

TokenTracker massively overcounts Codex Cache Read when a subagent rollout file contains the parent thread history copied by a thread_spawn fork.

This is separate from the local dashboard refresh problem in #67. Refreshing more reliably makes the inflated number visible sooner, but does not correct the accounting.

Observed behavior

A single multi-agent workflow produced six subagent rollout files. Each file replayed roughly 4,300 parent token_count records before the child actually began.

Sanitized evidence from one local-only investigation:

Measurement	Value
TokenTracker reported cached input	3,286,382,336
Corrected estimate after excluding fork replay	27,568,640
Approximate inflation	120x
Replayed cached tokens across six child files	3,295,807,232
Share caused by replayed history	99.17%

Root cause

A spawned Codex subagent rollout begins with:

session_meta.payload.source.subagent.thread_spawn

The file then contains copied parent history, followed by a deterministic boundary:

inter_agent_communication_metadata

TokenTracker's rollout parser currently accepts and accumulates every token_count record. It does not inspect session_meta.source.subagent.thread_spawn, and its prefilter skips inter_agent_communication_metadata. As a result, copied parent counters are treated as newly consumed tokens in every child file.

In all six inspected rollouts, every replayed parent token_count record was before the boundary and every genuine child record was after it.

Expected behavior

For a rollout identified as a subagent thread_spawn:

1. Do not ingest replayed parent token_count history.
2. Begin accounting only after inter_agent_communication_metadata.
3. Reset the cumulative-token baseline at that boundary.
4. If the boundary is not present yet because the file is partially written, fail closed: do not ingest replay and do not advance the cursor beyond data that can be safely retried.
5. Continue handling ordinary non-subagent rollouts exactly as today.

Avoid time-based heuristics; the rollout provides explicit structural markers.

Suggested implementation

• Detect session_meta.payload.source.subagent.thread_spawn at the start of a rollout.
• Preserve the boundary record in the initial line prefilter so the parser can switch from replay mode to child mode.
• Reset per-file cumulative totals after the boundary.
• Add regression fixtures for:
  • normal rollout;
  • complete subagent fork;
  • partially written fork without boundary;
  • resumed parse after the boundary appears;
  • multiple subagents forked from the same parent.
• Add an idempotent one-time migration/rebuild for affected Codex queue and project aggregates so already-inflated local history is corrected without altering other providers.

Privacy and deployment requirement

The fix, migration, tests, and normal operation must remain strictly local-only. No rollout contents, token records, project metadata, telemetry, diagnostics, or derived usage data may be sent to any cloud or third-party service.

Acceptance tests should include a no-network run that proves parsing and migration complete using only local files and local storage.

Workaround vs. fix

Reducing the context passed to subagents (fork_turns=none or a small explicit number plus a compact brief) reduces actual token usage. However, TokenTracker still needs this parser fix because existing and future forked rollout files must be accounted for correctly.

[pitimon]

Resolved across two merged PRs.

#77 — parser fix (23fc2d1). A Codex subagent fork replays the parent thread's cumulative token_count history before an inter_agent_communication_metadata boundary, then the genuine child turns. The parser counted the replayed parent deltas as the child's consumption. Root-caused against real local rollouts: the cumulative total_token_usage counter is continuous across the boundary, so the fix is baseline-tracking — advance the delta baseline through the replay without attributing, flip accounting on at the boundary. Fork lifecycle is persisted in the per-file cursor; a partial fork (no boundary yet) fails closed. Measured on 153 local fork rollouts: Cache Read 8,497,618,432 → 204,393,984 (~41.6x inflation removed). 9 regression tests.

#78 — one-time rebuild (b5266a9). The parser fix only corrects go-forward parsing (already-ingested files sit at offset = EOF and are never re-read). migrateCodexForkOvercountRebuild (run-once, source=codex/every-code scoped) drops the file cursors and retracts the inflated hourly buckets so the same-run re-parse rebuilds child-only totals; project aggregates are reset conservatively (accumulator only, no zeroing) so orphaned/moved-repo history is never lost. Strictly local — the acceptance test proves zero network calls. 4 tests.

Both structural markers (thread_spawn, inter_agent_communication_metadata) are pinned as a Codex format contract in the parser and tests. No time-based heuristics were used.

To apply the correction: run tokentracker sync once — the migration runs automatically and the dashboard's Codex Cache Read drops to the corrected value.