Usage cost report understates real AI cost by orders of magnitude: input/cache tokens are never counted and most day buckets are dropped

[atomantic]

Problem / Goal

The Est. API Cost Report on /devtools/usage reports numbers that are wrong by two to three orders of magnitude, so the page is not usable for its stated purpose ("what would this usage have cost under API billing").

Measured on a development install (rounded):

Number	Value
report.totals.estimatedCost for ?period=all	≈ $1.6
Legacy blended estimatedCost in the same response	≈ $258
All-time totalTokens.input (3.2k sessions, ~5 months)	≈ 98K
All-time totalTokens.output	≈ 17.2M
Sum of all per-day byProvider.tokensIn (what the report actually sums)	≈ 98K
Sum of all per-day byProvider.tokensOut	≈ 75K — i.e. 0.4% of the recorded 17.2M
Real input volume of one Claude Code session in this repo (from the CLI's own transcript)	3.54M cache-read + 288K cache-write + 24K output

That last row is the headline: a single agent session's real input volume is roughly 36× larger than PortOS's entire all-time recorded input token count, and PortOS prices the whole install's history at $1.6 while one session alone is worth ≈$4 at Opus rates.

Two numbers in the same API response differ by 160×, and the page renders the smaller one as the prominent bold figure.

Context

Where the data comes from today

Everything the report aggregates is written by server/services/usage.js (recordSession / recordMessages), fed from exactly two call sites:

• server/services/bootstrap.js:155-166 — the AI Toolkit onRunCreated / onRunCompleted hooks
• server/services/agentRunTracking.js:65 and :146-152 — CoS agent runs

Both estimate tokens the same way (server/lib/contextBudget.js, CHARS_PER_TOKEN = 4):

const estimatedTokens = estimateTokens(output);                     // output text length / 4
const inputTokens = estimateTokensFromChars(metadata.promptLength); // initial prompt length / 4

The report itself is buildUsageReport() in server/services/usage.js:365-483, surfaced by GET /api/usage (server/routes/usage.js:14-21) and rendered by CostReportTable in client/src/pages/UsagePage.jsx:192-257.

Root causes, in descending order of impact

1. Input tokens are estimated from the initial prompt only — the dominant cost driver is ~0.
metadata.promptLength is the length of the task description handed to the CLI. For an agentic run the real input is the cumulative per-turn context: system prompt + full conversation replay + every tool result, re-sent on every turn. Across 3.2k recorded sessions this install recorded ≈98K input tokens total — about 30 input tokens per session. A real session measures in the millions. Since input (and especially cache-read) volume is what actually drives API cost for agentic CLI use, understating it to ~0 alone accounts for most of the gap.

2. Cache-read / cache-write tokens are neither captured nor priced.
server/lib/modelPricing.js documents this as a deliberate simplification ("prompt-caching, batch, and long-context tier discounts are deliberately ignored; PortOS does not capture cache hits"). The header comment reasons about it as if ignoring caching makes the estimate high. In practice cache reads are >90% of input volume for agentic CLI runs, so omitting the tier entirely — while also not counting the tokens at all — makes the estimate catastrophically low. Ground truth for one session: 3.54M cache-read, 288K cache-write, 76 raw input.

3. The report silently drops most of the recorded history.
buildUsageReport skips any day bucket without a byProvider split (usage.js:409: if (!day?.byProvider) continue;). The per-provider/per-model split was added later, so on this install only 15 of 91 day buckets have it. The other 76 days — carrying ≈17.1M of the 17.2M recorded output tokens — contribute nothing to the report, even at ?period=all. report.breakdownSince discloses the cutoff in small print, but the headline dollar figure is presented without qualification.

4. Output tokens are derived from captured stdout, which for TUI providers is a repainted screen.
estimateTokens(output) measures the run's captured stdout. For TUI-driven providers that buffer is the ANSI-repainted terminal screen, not a transcript (see the tuiUsageScrape.js header and the "TUI raw.txt has ~no newlines" behavior) — so it is neither a count of generated tokens nor consistently over- or under-stated. The recorded per-message averages bear this out: codex averages ~30K output tokens per recorded message while claude-code averages ~1.2K.

5. An undefined provider row.
bootstrap.js's onRunCompleted calls recordMessages(metadata.providerId, …) with no guard, unlike agentRunTracking.completeAgentRun which gates on metadata.providerId && metadata.model. Runs that reach completion without a provider id create a literal byProvider['undefined'] bucket, which renders as a provider row named "undefined" in the cost table (≈14 messages / ≈1.4K tokens on this install).

6. totalToolCalls is permanently 0.
recordToolCalls() has no caller anywhere in the repo except the unused POST /api/usage/tokens-adjacent route (server/routes/usage.js:62); no client or service posts to it. The "Tool Calls" stat tile on the Usage page therefore always reads 0. The same is true of recordTokens() — the route exists, nothing calls it.

7. Sessions are recorded on run start, tokens only on clean exit.
recordSession fires in onRunCreated; recordMessages only fires on exitCode === 0. This install shows 3,247 sessions vs 2,067 messages — roughly a third of runs contribute a session count but zero tokens, so per-session averages in the report are biased low on top of everything else.

Ground truth is already on disk

Both major providers write real, per-message token counts that PortOS can read without spending a single token:

• Claude Code — ~/.claude/projects/<cwd-slug>/<session>.jsonl, one JSON object per message. Assistant messages carry message.usage = { input_tokens, cache_creation_input_tokens, cache_read_input_tokens, output_tokens } plus message.model, and every line carries cwd, timestamp, and sessionId. The project directory name is the slugified cwd (/ and . → -).
• Codex — ~/.codex/sessions/YYYY/MM/DD/rollout-<ts>-<uuid>.jsonl. Line 1 is a session_meta payload with id, cwd, cli_version, originator; later lines carry total_token_usage = { input_tokens, cached_input_tokens, output_tokens, reasoning_output_tokens, total_tokens }.

PortOS run metadata already records workspacePath, startTime, endTime, providerId, and model (data/runs/<id>/metadata.json), which is everything needed to correlate a run to its provider-native transcript. PortOS does not capture the CLI's own session id today.

Prior art for reading provider-native surfaces lives in server/services/claudeCodeUsage.js and server/services/providerUsage.js (subscription quota — a different concern; this issue is about PortOS's own per-run accounting in services/usage.js).

Proposed approach

Two phases. Phase 1 is independently shippable and fixes the largest single distortion without any new data source — ship it first, in its own PR.

Phase 1 — stop the report from lying (aggregation + hygiene)

1. Count legacy day buckets. In buildUsageReport, when a day bucket in range has no byProvider split, fold its flat { sessions, messages, tokens } into a synthetic provider row (id legacy, name Pre-breakdown (legacy), priced at the FALLBACK_RATES provider default via resolveModelRates(null, null)) instead of continue-ing. Keep breakdownSince meaning "per-provider detail starts here", and mark the legacy row rateMatch: 'fallback' so the UI's ~ approximate marker applies.
2. Reconcile the two cost numbers. Drop LEGACY_BLENDED_RATES / the top-level estimatedCost field from getUsageSummary, or keep it but stop shipping two contradictory numbers in one payload. Decision: keep the field for back-compat but recompute it from report.totals.estimatedCost over an unbounded range, so a consumer reading either one gets the same answer.
3. Guard the undefined provider. Gate bootstrap.js's onRunCompleted on metadata.providerId the way agentRunTracking.completeAgentRun already does, and have providerDayBucket refuse a nullish providerId (attribute to unknown with a real display name rather than the string "undefined"). Add a one-off migration in scripts/migrations/ that renames any existing byProvider['undefined'] key to unknown.
4. Fix or remove the dead counters. totalToolCalls reads 0 forever. Decision: wire it — increment from the same completion hooks using the tool-call count already parsed for run output where available, and when a provider yields no count, drop the "Tool Calls" tile rather than render a permanent 0.
5. Honest disclosure in the UI. The footnote in UsagePage.jsx:396-401 currently says token counts are "partially estimated". Replace with a specific, accurate statement of what is and is not counted (per-turn context, cache tiers), and surface breakdownSince as a visible caveat next to the headline figure rather than buried in the footnote.

Phase 2 — ingest provider-native ground truth

6. New service server/lib/providerTranscriptUsage.js (pure parsing helpers; barrel + README row per the Module Organization rule) with a reader per provider family:
   • parseClaudeTranscript(jsonlText) → { sessionId, cwd, model, messages, tokensIn, cacheReadTokens, cacheWriteTokens, tokensOut, firstTs, lastTs }
   • parseCodexRollout(jsonlText) → the same shape from session_meta + the final total_token_usage
     Both tolerate partial/truncated files (a session still being written) and unknown fields.
7. New service server/services/usageReconciler.js that, for a completed run, locates the matching transcript by slugify(workspacePath) + a [startTime, endTime] timestamp-overlap window, and records real counts via a new recordRunUsage({ providerId, model, tokensIn, tokensOut, cacheReadTokens, cacheWriteTokens }). When no transcript matches (provider has none, or the window is ambiguous), fall back to today's estimate and stamp the bucket source: 'estimate' so the report can distinguish measured from estimated rows.
8. Extend the day-bucket shape with cacheReadTokens / cacheWriteTokens / source, additively (absent = 0 / 'estimate'), so old buckets keep working. Bump the storage schemaVersion and ship a migration per the root CLAUDE.md distribution rules; keep data.reference/ seeds in sync.
9. Price the cache tiers in server/lib/modelPricing.js: add cacheReadPer1M / cacheWritePer1M to the rate shape (Anthropic: 0.1× input for reads, 1.25× input for 5-minute writes; OpenAI/xAI/Gemini per their published cached-input rates) and extend estimateCostUsd to take the two new buckets. Correct the header comment, which currently reasons about caching as if ignoring it inflates the estimate.
10. UI: add Cache Read / Cache Write columns to CostReportTable (collapsed into a tooltip on narrow viewports per the mobile-responsive rule), and mark estimate-sourced rows distinctly from measured rows.
11. Backfill: a one-shot, user-triggered endpoint + button that re-reads the on-disk transcripts for the retained day-bucket window and replaces estimates with measured counts. Explicitly not run at boot — reading local JSONL makes no provider call, but a from-zero bulk pass still belongs behind an explicit user action per the AI Provider Usage Policy's background-pre-generation pattern.

Acceptance criteria

• GET /api/usage?period=all returns a report.totals.estimatedCost that accounts for every day bucket in data/usage.json, including buckets with no byProvider split — verified by a unit test with a fixture mixing legacy flat buckets and modern split buckets.
• report.totals.estimatedCost (unbounded range) and the top-level estimatedCost field agree to within rounding; no response ships two figures that differ by more than 1%.
• No provider row is ever named undefined; a run completing without a provider id attributes to an unknown row, and a migration renames existing byProvider['undefined'] keys.
• The "Tool Calls" tile either shows a non-zero count for installs with tool-using runs, or is removed.
• parseClaudeTranscript and parseCodexRollout unit tests cover: a complete session, a truncated/mid-write session, a session with zero assistant messages, and unknown/extra fields.
• For a run whose transcript is found, the recorded tokensIn + cacheReadTokens + cacheWriteTokens + tokensOut match the transcript's own sums exactly (no estimation), asserted in a test against a checked-in redacted fixture.
• estimateCostUsd prices cache-read and cache-write at their own per-1M rates; a test asserts an Anthropic model's cache-read cost is 10% of its input rate.
• Day buckets carry source: 'measured' | 'estimate'; the cost table visually distinguishes the two, and a run with no matching transcript still records (falling back to the estimate) rather than recording nothing.
• The day-bucket shape change ships with a migration in scripts/migrations/ and a schemaVersion bump; a pre-change data/usage.json loads without error on the new code.
• The UI footnote states specifically what is counted (per-turn context, cache tiers) rather than "partially estimated", and breakdownSince is visible next to the headline figure.
• The backfill is reachable only from an explicit user action — no boot-time or scheduled invocation.
• cd server && npm test and cd client && npm test pass.

Out of scope

• The Subscription Usage section at the top of the page (GET /api/usage/providers, providerUsage.js, claudeCodeUsage.js, tuiUsageScrape.js) — plan rate-limit meters are a separate surface and are not implicated here.
• Reading usage from provider billing APIs or web dashboards. This issue stays local: on-disk transcripts only.
• Retroactively reconstructing per-turn token counts for runs whose provider writes no transcript (ollama, LM Studio, and any local/free provider) — those stay estimated and priced at $0 via isFreeProvider.
• Changing the 400-day daily→monthly rollup retention policy.
• Any change to how runs are executed or how output is captured.