feat(web): usage page showing what your agents actually cost - #5619
feat(web): usage page showing what your agents actually cost#5619t3dotgg wants to merge 4 commits into
Conversation
Adds a Usage page under settings that reports token spend across every connected environment. Cost comes from the session logs Claude Code and Codex already write to disk rather than T3 Code's event log. The event log only sees threads T3 Code drove and carries no cache-write counts for Claude, which makes exact cost impossible to derive from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:trusted
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
There was a problem hiding this comment.
Effect Service Conventions
Three findings, all in the new usage service and its call sites. The service itself is otherwise conventional (inline Context.Service interface, dependencies acquired with yield* from the environment, no hidden runtime).
apps/server/src/usage/UsageService.ts— layer exported asUsageServiceLayerwith construction inlined; convention ismake+export const layer.apps/server/src/server.ts— named import of the layer erases the module namespace used for every other service in this file.apps/web/src/state/usage.ts—UsageFetchErroris aData.TaggedErrorcarrying only an opaquecause, with no structural attributes or derivedmessage.
Posted via Macroscope — Effect Service Conventions
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| continue; | ||
| } | ||
| seen.add(key); | ||
| } |
There was a problem hiding this comment.
Claude duplicates without message id
Medium Severity
Claude deduplication runs only when message.id is a non-empty string. Assistant rows without an id still bill on every file repeat, contradicting the stated dedup-on-message-id rule and risking overcount when ids are absent.
Reviewed by Cursor Bugbot for commit 98e451c. Configure here.
There was a problem hiding this comment.
Note
🤖 Claude Opus 5 responding on behalf of Theo
Leaving as is. Claude always writes message.id on assistant records: 0 of 22,161 rows across a 406-file sample of ~/.claude/projects were missing it.
The guard exists so a malformed row degrades to being counted rather than throwing, which is the safer direction for a reporting page. Falling back to a synthetic key (timestamp plus token counts) would risk collapsing two genuinely identical turns into one, which is a worse failure than the one it prevents.
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
|
|
||
| const hourRows = yield* sql<{ hour: unknown; count: unknown }>` | ||
| SELECT | ||
| CAST(strftime('%H', requested_at) AS INTEGER) AS hour, |
There was a problem hiding this comment.
🟡 Medium usage/activityUsage.ts:86
turnsByHour places each turn in the wrong hour bucket on any non-UTC host. The SQL uses strftime('%H', requested_at) on stored ISO-8601 timestamps ending in Z, so SQLite interprets them as UTC and the result is the UTC hour rather than local time. The contract documents these buckets as local-time hours, so the histogram is shifted (and near midnight turns can land on the wrong day) for every user outside UTC. Add the 'localtime' modifier, e.g. strftime('%H', requested_at, 'localtime'), so the buckets reflect local working hours.
| CAST(strftime('%H', requested_at) AS INTEGER) AS hour, | |
| CAST(strftime('%H', requested_at, 'localtime') AS INTEGER) AS hour, |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/usage/activityUsage.ts around line 86:
`turnsByHour` places each turn in the wrong hour bucket on any non-UTC host. The SQL uses `strftime('%H', requested_at)` on stored ISO-8601 timestamps ending in `Z`, so SQLite interprets them as UTC and the result is the UTC hour rather than local time. The contract documents these buckets as local-time hours, so the histogram is shifted (and near midnight turns can land on the wrong day) for every user outside UTC. Add the `'localtime'` modifier, e.g. `strftime('%H', requested_at, 'localtime')`, so the buckets reflect local working hours.
|
|
||
| const messageId = messageRecord["id"]; | ||
| if (typeof messageId === "string" && messageId.length > 0) { | ||
| const key = `${messageId}:${String(entry["requestId"] ?? "")}`; |
There was a problem hiding this comment.
🟡 Medium usage/localAgentUsage.ts:240
scanClaude deduplicates records by a key of messageId:requestId, but a resumed or forked session copies the same assistant message into a new file with a different requestId. Because the key differs, both copies are billed — inflating tokens, messages, and cost, which is the exact overcount this deduplication step exists to prevent. Key seen by messageId alone.
| const key = `${messageId}:${String(entry["requestId"] ?? "")}`; | |
| const key = messageId; |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/usage/localAgentUsage.ts around line 240:
`scanClaude` deduplicates records by a key of `messageId:requestId`, but a resumed or forked session copies the same assistant message into a new file with a *different* `requestId`. Because the key differs, both copies are billed — inflating tokens, messages, and cost, which is the exact overcount this deduplication step exists to prevent. Key `seen` by `messageId` alone.
| export type UsageProvider = typeof UsageProvider.Type; | ||
|
|
||
| /** An ISO date (YYYY-MM-DD) in the host's local timezone. */ | ||
| export const UsageDate = TrimmedNonEmptyString; |
There was a problem hiding this comment.
🟡 Medium src/usage.ts:19
UsageDate is documented as an ISO YYYY-MM-DD date but is defined as TrimmedNonEmptyString, so it accepts any non-empty string like sinceDate=zzz or sinceDate=2026-99-99. These malformed values pass validation and are used lexicographically downstream and appended with T00:00:00.000Z, producing an incorrect or empty usage snapshot instead of a 400 error. Consider constraining UsageDate with a pattern or date schema so only valid YYYY-MM-DD strings are accepted.
| export const UsageDate = TrimmedNonEmptyString; | |
| +/** An ISO date (YYYY-MM-DD) in the host's local timezone. */ | |
| +export const UsageDate = TrimmedNonEmptyString.pipe( | |
| + Schema.pattern(/^\d{4}-\d{2}-\d{2}$/), | |
| +); |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/contracts/src/usage.ts around line 19:
`UsageDate` is documented as an ISO `YYYY-MM-DD` date but is defined as `TrimmedNonEmptyString`, so it accepts any non-empty string like `sinceDate=zzz` or `sinceDate=2026-99-99`. These malformed values pass validation and are used lexicographically downstream and appended with `T00:00:00.000Z`, producing an incorrect or empty usage snapshot instead of a 400 error. Consider constraining `UsageDate` with a pattern or date schema so only valid `YYYY-MM-DD` strings are accepted.
| planType: typeof planType === "string" && planType.length > 0 ? planType : undefined, | ||
| usedPercent, | ||
| windowMinutes: toInt(primaryRecord["window_minutes"]), | ||
| resetsAt: typeof resetsAt === "number" ? new Date(resetsAt * 1000).toISOString() : undefined, |
There was a problem hiding this comment.
🟡 Medium usage/localAgentUsage.ts:457
parseRateLimit calls new Date(resetsAt * 1000).toISOString() when resets_at is any number, including NaN or Infinity. These produce an invalid Date, so toISOString() throws RangeError, aborting the entire usage scan for a single malformed rate-limit record. Other malformed entries in this file are silently skipped. Consider guarding resets_at with Number.isFinite before constructing the date.
| resetsAt: typeof resetsAt === "number" ? new Date(resetsAt * 1000).toISOString() : undefined, | |
| resetsAt: typeof resetsAt === "number" && Number.isFinite(resetsAt) ? new Date(resetsAt * 1000).toISOString() : undefined, |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/usage/localAgentUsage.ts around line 457:
`parseRateLimit` calls `new Date(resetsAt * 1000).toISOString()` when `resets_at` is any number, including `NaN` or `Infinity`. These produce an invalid `Date`, so `toISOString()` throws `RangeError`, aborting the entire usage scan for a single malformed rate-limit record. Other malformed entries in this file are silently skipped. Consider guarding `resets_at` with `Number.isFinite` before constructing the date.
| return found; | ||
| } | ||
|
|
||
| async function* readJsonLines(path: string): AsyncGenerator<Record<string, unknown>> { |
There was a problem hiding this comment.
🟡 Medium usage/localAgentUsage.ts:116
readJsonLines lets stream errors propagate out of the async iterator, so a single unreadable .jsonl file rejects the entire readLocalAgentUsage scan and discards all otherwise-readable usage data. This happens because NodeFS.createReadStream emits an error event that is not caught before iterating NodeReadline.createInterface. Consider attaching an error listener that closes the generator so the scan skips the bad file instead of aborting.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/usage/localAgentUsage.ts around line 116:
`readJsonLines` lets stream errors propagate out of the async iterator, so a single unreadable `.jsonl` file rejects the entire `readLocalAgentUsage` scan and discards all otherwise-readable usage data. This happens because `NodeFS.createReadStream` emits an `error` event that is not caught before iterating `NodeReadline.createInterface`. Consider attaching an `error` listener that closes the generator so the scan skips the bad file instead of aborting.
ApprovabilityVerdict: Needs human review 6 blocking correctness issues found. New feature adding a usage/cost reporting page with new API endpoint, service layer, log scanning, and UI components. Multiple unresolved review comments exist, including a high-severity unbounded cache issue and medium-severity bugs in deduplication and cost calculation logic. You can customize Macroscope's approvability policy. Learn more. |
Per-environment auth, correct Codex fast-mode pricing, and several accuracy fixes raised by review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Note 🤖 Claude Opus 5 responding on behalf of Theo Pushed fixes for the review findings. Two were real breakage of the multi-environment story and are worth calling out. Per-environment auth. Every environment was being fetched through Stale connection reads. Also fixed:
Effect conventions from the other bot are addressed too: Two findings I pushed back on with reasons, both in the threads above. Still not verified in a running browser. Preview automation times out in my environment, so the React rendering remains typechecked but not visually confirmed. Server side is exercised against real data: 10 unit tests, plus all five activity queries run against the real projections database. |
| const files = await collectJsonlFiles(root); | ||
| let recordsRead = 0; | ||
| let latest: UsageRateLimitWindow | undefined; | ||
|
|
||
| for (const file of files) { | ||
| if (await isStale(file, sinceDate)) continue; |
There was a problem hiding this comment.
🟡 Medium usage/localAgentUsage.ts:334
scanCodex skips stale files via isStale but still uses files.length for both filesScanned and sessions, so the returned report overcounts. A 30-day query that skips hundreds of older files still reports them all as scanned sessions, inflating sources[].filesScanned, sources[].sessions, and LocalUsageResult.sessions. Track the count of files actually scanned separately from the count of files found.
const files = await collectJsonlFiles(root);
+ let filesScanned = 0;
let recordsRead = 0;
let latest: UsageRateLimitWindow | undefined;
for (const file of files) {
if (await isStale(file, sinceDate)) continue;
+ filesScanned += 1;🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/usage/localAgentUsage.ts around lines 334-339:
`scanCodex` skips stale files via `isStale` but still uses `files.length` for both `filesScanned` and `sessions`, so the returned report overcounts. A 30-day query that skips hundreds of older files still reports them all as scanned sessions, inflating `sources[].filesScanned`, `sources[].sessions`, and `LocalUsageResult.sessions`. Track the count of files actually scanned separately from the count of files found.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8afa8ee. Configure here.
Adds an end-to-end usage endpoint test and a T3CODE_AGENT_LOG_HOME override so agent log reads can be isolated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Note 🤖 Claude Opus 5 responding on behalf of Theo Round two pushed. Both new findings were real. Unbounded cache. Latched fast pricing. Also added the end-to-end coverage I owed, since a browser is not available in my environment:
That first test earned its keep immediately. It failed on the initial run by returning $13,487 of real spend, which proved the endpoint works end to end but also exposed that the reader always reads the OS home regardless of the server's base directory. Added a 142 server tests pass, typecheck and lint clean. On the approvability verdict: "needs human review" is the right call and not something I should try to clear. This is a deliberately WIP PR for @t3dotgg on a ~1,500 line new feature, and the two open threads are ones I pushed back on with evidence rather than fixed. |
|
#5652 is way better |
|
Wrong pr, correct analysis #5684 |
1 participant
WIP. Server, contracts and UI are done and green, but the page has not been eyeballed in a running app yet. Filing for review on the approach.
Problem
There was no way to see what any of this costs. The obvious source, T3 Code's own event log, turns out to be the wrong one: it only sees threads T3 Code drove, and it carries no cache-write counts for Claude at all (0 of 34,650 rows on my machine). Cost simply is not derivable from it.
On my real data the event log implies about $1.9k. The actual number is $16.2k. It was undercounting by roughly 8x.
Solution
Read the session logs Claude Code and Codex already write to disk (
~/.claude/projects,~/.codex/sessions), which carry exact per-message token counts including the cache buckets. This is the same sourceccusageuses, and its adapters were the reference for the parsing rules.The behavioral half of the page (skills, tools, subagents, diffs, turns by hour) still comes from our event log, since the agent logs know nothing about those. The two are complements.
Each environment reports its own host, and the client fans out and merges, so a remote or SSH environment contributes its own totals. An environment that is offline degrades to a row instead of blanking the page.
Three rules that carry the correctness
Tests cover each of those plus the unpriced-model and no-logs cases.
Also worth knowing
Codex writes its remaining quota into every session log, so the page can show real rate-limit headroom (plan type, percent used, reset date). We currently drop that on ingestion.
Known gaps
ccusageat about 7%, with the residual in models proxied through another CLI. A model with no entry shows tokens and is markedunpricedrather than counted as free.Verification
typecheckandlintclean, 7 new tests pass, and the full 122-test server suite still passes (this touchesmakeRoutesLayer).Built with Claude Opus 5 (1M context) in T3 Code.
Note
Medium Risk
New authenticated read API scans local filesystem and SQL; cost figures depend on parsing/pricing heuristics, but failures degrade rather than blocking core flows.
Overview
Adds usage reporting so spend and activity are visible per host instead of inferring cost from T3’s event log (which misses cache writes and most agent sessions).
Server: Authenticated
GET /api/usage/snapshot(orchestration read scope) builds anEnvironmentUsageSnapshotby merging local Claude/Codex JSONL session logs (dedup, Codex cumulative deltas, offline pricing table) with SQL projection activity (tools, skills, turns-by-hour UTC, checkpoint line churn).UsageServicecaches complete snapshots ~60s persinceDatewindow and degrades to zeros on partial failures.Client: New Settings → Usage page fans out to every connected environment via
fetchEnvironmentUsageSnapshot(per-env credentials, 30s timeout), merges totals client-side, and shows spend/tokens, daily charts, models, per-environment rows, and activity breakdowns with 7d/30d/90d windows.Contracts: Shared
usage.tsschemas and HTTP API group wire the snapshot shape end-to-end.Reviewed by Cursor Bugbot for commit ae37ed7. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add a usage dashboard page showing agent spend, token usage, and activity per environment
/settings/usagepage (UsageSettings.tsx) with a daily spend chart, per-model and per-environment tables, turns-by-hour heat strip, and top tools/skills/subagents, with selectable 7/30/90-day windows.useUsagehook (state/usage.ts) that concurrently fetches usage snapshots across all connected environments, merges them into aggregated totals, and tracks loading state./api/usage/snapshotGET endpoint (http.ts) requiring orchestration read scope, backed by aUsageServicewith 60-second per-window caching.Macroscope summarized ae37ed7.