feat(events): emit 5-way token breakdown + context-window utilization in message_complete

[byapparov]

Summary

• 5-way token breakdown in message_complete: expands tokens from an opaque info.tokens passthrough to an explicit object — input / output / reasoning / cache.read / cache.write — mirroring upstream LLM.Usage. The data was already captured in MessageV2.Assistant.tokens via StepFinishPart accumulation; this change surfaces all fields explicitly.
• Context-window utilization added as context: { used, limit, ratio } where used = input + cache.read, limit comes from Provider.getModel() → model.limit.context (models.dev registry), and ratio = used / limit. Emits null when the model's context limit is not known (unregistered custom endpoint).
• cost: 0 trap avoided: info.cost is kept as-is — it accumulates real per-step cost from StepFinishPart. The new upstream step.ended event emits cost: 0 (reconciled later by a projector); we do not touch that path.
• EVENTS.md updated with the extended schema, field-by-field documentation, and non-overlap invariant note.

How the cache split was already captured

MessageV2.StepFinishPart (the legacy step-finish message part) already has tokens: { input, output, reasoning, cache: { read, write } }. The assistant message info.tokens is accumulated from these step parts — cache split included. No new provider-level capture was needed; we just stop dropping it in the emit call.

Context limit source

Provider.getModel(providerID, modelID) returns the model record from the models.dev registry, which has model.limit.context. The lookup is wrapped in a .catch(() => null) so an unknown model (custom endpoint) gracefully emits context: null rather than throwing.

Test plan

• TDD: wrote failing test in packages/cli/test/cli/usage-token-breakdown.test.ts (8 cases) — confirmed RED before implementation
• Implemented changes in packages/cli/src/cli/cmd/run.ts
• bun test test/cli/usage-token-breakdown.test.ts — 8/8 GREEN
• bun test test/cli/ — 77/77 GREEN (no regressions)
• bun run typecheck — clean
• Pre-push hook ran bun turbo typecheck across all 5 packages — 6/6 tasks successful

Closes #86

🤖 Generated with Claude Code

https://claude.ai/code/session_0187MsfK1upr6K2BKVbmaebQ

[aictrl-dev]

Code review

Verdict: Address the major findings before merging. · 🔴 0 · 🟠 1 · 🟡 1 · ⚪ 0 · 0/2 resolved

• 🟠 packages/cli/src/cli/cmd/run.ts:488-494 — limit:0 yields Infinity ratio, breaks null contract
• 🟡 packages/cli/test/cli/usage-token-breakdown.test.ts:63 — Tests grep source text instead of running emit path

🤖 Fix all 2 open findings with your agent

Fix the following code review findings on aictrl-dev/cli PR #87 (head branch).
Run the relevant tests/linters after each change.

1. packages/cli/src/cli/cmd/run.ts:488-494 — limit:0 yields Infinity ratio, breaks null contract
   Detail: The guard `contextLimit != null` does not catch the case where the model's context limit is `0`. `Provider.Model.limit.context` is a required `z.number()` (provider.ts:703), and config-defined custom models without a limit default `context` to `0` (provider.ts:929: `context: model.limit?.context ?? existingModel?.limit?.context ?? 0`). For such models `Provider.getModel` succeeds and returns `context: 0`, so `contextLimit` is `0` (not `null`), the guard passes, and `ratio = contextUsed / 0` evaluates to `Infinity` (or `NaN` when `contextUsed` is also `0`). `JSON.stringify` then serializes `Infinity`/`NaN` as `null` *inside* the context object, yielding `{ used: <N>, limit: 0, ratio: null }` instead of the documented top-level `null`. EVENTS.md explicitly promises: "null — emitted when the model's context limit is not known (e.g. unregistered custom endpoint)". This diverges from the documented contract and emits a misleading object for the exact scenario it claims to handle. Fix: also require `contextLimit > 0` before emitting the object.
   Suggested fix: Change the guard from `contextLimit != null` to `contextLimit != null && contextLimit > 0` so a context limit of `0` (config/custom models without a known limit) is treated as "unknown" and emits `null` as documented, avoiding `Infinity`/`NaN` in the ratio.
2. packages/cli/test/cli/usage-token-breakdown.test.ts:63 — Tests grep source text instead of running emit path
   Detail: The new tests assert behavior by reading `run.ts` as a string and matching regexes against the source text rather than exercising the emit path. Several assertions are loose enough to pass even if the logic is wrong:\n\n- `test("context.ratio is used / limit")` (line 63) uses `/ratio.*\\/|\\/.*ratio|used\\s*\\/\\s*limit|contextLimit/`. The `contextLimit` alternative matches the bare variable declaration on line 483, so the test stays green even if `ratio` is changed to a constant (e.g. `ratio: 0`).\n- `test("context.limit is sourced from Provider.getModel context limit")` (line 57) uses `/Provider\\.getModel|limit\\.context|contextLimit/` — any one of those tokens anywhere in the file passes; it does not verify the value is actually wired into `context.limit`.\n\nThese give a false sense of coverage: a refactor that renames a variable or removes the real computation but leaves a stray token would not be caught. Prefer asserting against actual emitted JSON (e.g. stub `Provider.getModel` and capture `process.stdout` writes from `emit("message_complete", …)`) so the 5-way `tokens` and `context` shapes and the `used = input + cache.read` / `ratio = used/limit` computations are verified at runtime.

📋 Out-of-diff findings (2)

Sev	Location	Finding
🟠	packages/cli/src/cli/cmd/run.ts:488-494	limit:0 yields Infinity ratio, breaks null contract
🟡	packages/cli/test/cli/usage-token-breakdown.test.ts:63	Tests grep source text instead of running emit path

Reviewed 3 files · 0 inline · view all 2 findings ↗

---

aictrl · AI code review for fast-moving teams · aictrl.dev

[byapparov]

Review response — PR #87

Triaged 2 findings (🟠 1, 🟡 1). Both verified TRUE; both fixed.

Issues addressed (pushed to this PR)

• limit:0 yields Infinity ratio, breaks null contract — packages/cli/src/cli/cmd/run.ts:488: Verified TRUE. provider.ts:929 confirms custom models default context to 0. The old contextLimit != null guard passes for 0, so ratio = used / 0 = Infinity, which JSON.stringify serialises as null inside the object — breaking the documented top-level null contract in EVENTS.md line 113. Fixed by extracting buildContextWindow(limit, used) which returns null when limit == null || limit <= 0. (commit 7bbf2045d)

• Tests grep source text instead of running emit path — packages/cli/test/cli/usage-token-breakdown.test.ts:63: Verified TRUE. context.ratio is used / limit matched the bare contextLimit variable declaration, not the computation — the test stayed green with ratio: 0 hardcoded. Fixed by replacing the 5 loose regex tests with 10 behavioural unit tests of the extracted buildContextWindow helper: null limit, zero limit (regression case), ratio math, JSON-serialisability, top-level-null contract. Source-text checks retained only for structural wiring that can't be covered by the pure helper. (commit 7bbf2045d)

Review claims verified false (no change needed)

(none — both findings were genuine)

Not addressed here

(none — all findings fixed)

[aictrl-dev]

🟡 ratio unclamped but documented as 0–1.

Suggested change

	limit: contextLimit,
	In buildContextWindow: `ratio: Math.min(1, contextUsed / contextLimit),` — or update EVENTS.md from "ratio (0–1)" to "ratio (0–1; may exceed 1 if usage exceeds the registered limit)".

🤖 Fix with your agent

Fix this code review finding (aictrl-dev/cli PR #87, packages/cli/src/cli/cmd/run.ts:102):

Problem: ratio unclamped but documented as 0–1
Detail: `buildContextWindow` computes `ratio: contextUsed / contextLimit` with no clamp, but EVENTS.md documents `ratio` as `(0–1)`. The code can legitimately produce `ratio > 1`: if `contextUsed` ever exceeds the registry limit (stale/lowered `model.limit.context` after a model update, or a provider whose actual billed prompt exceeds the registered window), `ratio` exceeds 1 and contradicts the documented range. Consumers relying on `0 ≤ ratio ≤ 1` (e.g. for a utilization bar) will break. Either clamp (`Math.min(1, used/limit)`) or relax the doc to note values may exceed 1 when usage exceeds the registered limit.
Suggested fix: In buildContextWindow: `ratio: Math.min(1, contextUsed / contextLimit),` — or update EVENTS.md from "ratio (0–1)" to "ratio (0–1; may exceed 1 if usage exceeds the registered limit)".

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.

Why this matters

buildContextWindow computes ratio: contextUsed / contextLimit with no clamp, but EVENTS.md documents ratio as (0–1). The code can legitimately produce ratio > 1: if contextUsed ever exceeds the registry limit (stale/lowered model.limit.context after a model update, or a provider whose actual billed prompt exceeds the registered window), ratio exceeds 1 and contradicts the documented range. Consumers relying on 0 ≤ ratio ≤ 1 (e.g. for a utilization bar) will break. Either clamp (Math.min(1, used/limit)) or relax the doc to note values may exceed 1 when usage exceeds the registered limit.

  if (contextLimit == null || contextLimit <= 0) return null
  return {
    used: contextUsed,
    limit: contextLimit,
    ratio: contextUsed / contextLimit,
  }
}

[aictrl-dev]

🟡 Verify upstream tokens shape for all providers.

🤖 Fix with your agent

Fix this code review finding (aictrl-dev/cli PR #87, packages/cli/src/cli/cmd/run.ts:510):

Problem: Verify upstream tokens shape for all providers
Detail: The correctness of both `context.used` and the `reasoning` field hinges on the upstream `info.tokens` shape that this diff cannot show:
1. `contextUsed = tokens.input + tokens.cache.read` is correct only if `input` EXCLUDES cached tokens (Anthropic semantics). For OpenAI, raw `prompt_tokens` historically INCLUDES cached tokens — if the upstream `LLM.Usage` isn't normalized per-provider, `used` double-counts cache reads, and the EVENTS.md "a token is counted in exactly one bucket" guarantee silently fails for those providers.
2. `reasoning: info.tokens.reasoning` emits `undefined` (dropped by JSON.stringify → missing field) if any provider's `StepFinishPart` doesn't populate `reasoning`.
Please confirm the upstream normalization covers every provider before relying on these metrics; otherwise guard with `(info.tokens.reasoning ?? 0)` and verify `input` excludes cache for OpenAI.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.

Why this matters

The correctness of both context.used and the reasoning field hinges on the upstream info.tokens shape that this diff cannot show:

1. contextUsed = tokens.input + tokens.cache.read is correct only if input EXCLUDES cached tokens (Anthropic semantics). For OpenAI, raw prompt_tokens historically INCLUDES cached tokens — if the upstream LLM.Usage isn't normalized per-provider, used double-counts cache reads, and the EVENTS.md "a token is counted in exactly one bucket" guarantee silently fails for those providers.
2. reasoning: info.tokens.reasoning emits undefined (dropped by JSON.stringify → missing field) if any provider's StepFinishPart doesn't populate reasoning.
   Please confirm the upstream normalization covers every provider before relying on these metrics; otherwise guard with (info.tokens.reasoning ?? 0) and verify input excludes cache for OpenAI.

                const contextLimit = await Provider.getModel(info.providerID, info.modelID)
                  .then((m) => m.limit.context)
                  .catch(() => null)
                const contextUsed = tokens.input + tokens.cache.read
                const context = buildContextWindow(contextLimit, contextUsed)

[aictrl-dev]

🟡 .catch(()=>null) swallows non-not-found errors.

🤖 Fix with your agent

Fix this code review finding (aictrl-dev/cli PR #87, packages/cli/src/cli/cmd/run.ts:509):

Problem: .catch(()=>null) swallows non-not-found errors
Detail: `Provider.getModel(...).catch(() => null)` turns *every* rejection into the "unknown context" sentinel — not only the intended "model not registered" case. A transient registry fetch failure, a programming error in `getModel`, or a malformed `m.limit` shape would all silently degrade `message_complete.context` to `null` with zero signal, making regressions invisible. Consider catching only the expected not-found error type, or logging the unexpected error before falling back to `null`.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.

Why this matters

Provider.getModel(...).catch(() => null) turns every rejection into the "unknown context" sentinel — not only the intended "model not registered" case. A transient registry fetch failure, a programming error in getModel, or a malformed m.limit shape would all silently degrade message_complete.context to null with zero signal, making regressions invisible. Consider catching only the expected not-found error type, or logging the unexpected error before falling back to null.

                const contextLimit = await Provider.getModel(info.providerID, info.modelID)
                  .then((m) => m.limit.context)
                  .catch(() => null)
                const contextUsed = tokens.input + tokens.cache.read

[aictrl-dev]

⚪ Example ratio 0.049 ≠ emitted 0.04912.

🤖 Fix with your agent

Fix this code review finding (aictrl-dev/cli PR #87, EVENTS.md:91):

Problem: Example ratio 0.049 ≠ emitted 0.04912
Detail: The `message_complete` example shows `"context": { "used": 9824, "limit": 200000, "ratio": 0.049 }`, but `buildContextWindow` emits the unrounded float: `9824 / 200000 = 0.04912`. The example rounds to 3 decimals, which may mislead consumers who treat the doc as a byte-exact spec. Either note that `ratio` is an unrounded float, or pick an example whose `used/limit` is exact (e.g. used 9800 → 0.049).

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.

Why this matters

The message_complete example shows "context": { "used": 9824, "limit": 200000, "ratio": 0.049 }, but buildContextWindow emits the unrounded float: 9824 / 200000 = 0.04912. The example rounds to 3 decimals, which may mislead consumers who treat the doc as a byte-exact spec. Either note that ratio is an unrounded float, or pick an example whose used/limit is exact (e.g. used 9800 → 0.049).

    "cache": { "read": 8800, "write": 1024 }
  },
  "context": { "used": 9824, "limit": 200000, "ratio": 0.049 },
  "finish": "tool-calls"

[aictrl-dev]

⚪ JSDoc hardcodes provider.ts:929 (line rot).

🤖 Fix with your agent

Fix this code review finding (aictrl-dev/cli PR #87, packages/cli/src/cli/cmd/run.ts:88):

Problem: JSDoc hardcodes provider.ts:929 (line rot)
Detail: The `buildContextWindow` JSDoc references `` `context: 0` (provider.ts:929) ``. A hardcoded file:line citation will silently rot the moment `provider.ts` is edited, turning the comment into misleading guidance. Reference the behavior/symbol (e.g. "the provider's default `limit.context = 0` for unregistered custom models") instead of a line number.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.

Why this matters

The buildContextWindow JSDoc references `context: 0` (provider.ts:929). A hardcoded file:line citation will silently rot the moment provider.ts is edited, turning the comment into misleading guidance. Reference the behavior/symbol (e.g. "the provider's default limit.context = 0 for unregistered custom models") instead of a line number.

 * - `contextLimit` is `0`   — custom model without a registered limit defaults to
 *   `context: 0` (provider.ts:929). A zero limit would yield `Infinity`/`NaN` for
 *   ratio, which `JSON.stringify` serialises as `null` inside the object — diverging

[aictrl-dev]

Code review

Verdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 3 · ⚪ 3 · 0/6 resolved

• ⚪ EVENTS.md:91 — Example ratio 0.049 ≠ emitted 0.04912
• ⚪ packages/cli/src/cli/cmd/run.ts:88 — JSDoc hardcodes provider.ts:929 (line rot)
• 🟡 packages/cli/src/cli/cmd/run.ts:102 — ratio unclamped but documented as 0–1
• 🟡 packages/cli/src/cli/cmd/run.ts:509 — .catch(()=>null) swallows non-not-found errors
• 🟡 packages/cli/src/cli/cmd/run.ts:510 — Verify upstream tokens shape for all providers
• ⚪ packages/cli/test/cli/usage-token-breakdown.test.ts:217 — Source-substring tests are brittle

🤖 Fix all 6 open findings with your agent

Fix the following code review findings on aictrl-dev/cli PR #87 (head branch).
Run the relevant tests/linters after each change.

1. EVENTS.md:91 — Example ratio 0.049 ≠ emitted 0.04912
   Detail: The `message_complete` example shows `"context": { "used": 9824, "limit": 200000, "ratio": 0.049 }`, but `buildContextWindow` emits the unrounded float: `9824 / 200000 = 0.04912`. The example rounds to 3 decimals, which may mislead consumers who treat the doc as a byte-exact spec. Either note that `ratio` is an unrounded float, or pick an example whose `used/limit` is exact (e.g. used 9800 → 0.049).
2. packages/cli/src/cli/cmd/run.ts:88 — JSDoc hardcodes provider.ts:929 (line rot)
   Detail: The `buildContextWindow` JSDoc references `` `context: 0` (provider.ts:929) ``. A hardcoded file:line citation will silently rot the moment `provider.ts` is edited, turning the comment into misleading guidance. Reference the behavior/symbol (e.g. "the provider's default `limit.context = 0` for unregistered custom models") instead of a line number.
3. packages/cli/src/cli/cmd/run.ts:102 — ratio unclamped but documented as 0–1
   Detail: `buildContextWindow` computes `ratio: contextUsed / contextLimit` with no clamp, but EVENTS.md documents `ratio` as `(0–1)`. The code can legitimately produce `ratio > 1`: if `contextUsed` ever exceeds the registry limit (stale/lowered `model.limit.context` after a model update, or a provider whose actual billed prompt exceeds the registered window), `ratio` exceeds 1 and contradicts the documented range. Consumers relying on `0 ≤ ratio ≤ 1` (e.g. for a utilization bar) will break. Either clamp (`Math.min(1, used/limit)`) or relax the doc to note values may exceed 1 when usage exceeds the registered limit.
   Suggested fix: In buildContextWindow: `ratio: Math.min(1, contextUsed / contextLimit),` — or update EVENTS.md from "ratio (0–1)" to "ratio (0–1; may exceed 1 if usage exceeds the registered limit)".
4. packages/cli/src/cli/cmd/run.ts:509 — .catch(()=>null) swallows non-not-found errors
   Detail: `Provider.getModel(...).catch(() => null)` turns *every* rejection into the "unknown context" sentinel — not only the intended "model not registered" case. A transient registry fetch failure, a programming error in `getModel`, or a malformed `m.limit` shape would all silently degrade `message_complete.context` to `null` with zero signal, making regressions invisible. Consider catching only the expected not-found error type, or logging the unexpected error before falling back to `null`.
5. packages/cli/src/cli/cmd/run.ts:510 — Verify upstream tokens shape for all providers
   Detail: The correctness of both `context.used` and the `reasoning` field hinges on the upstream `info.tokens` shape that this diff cannot show:
1. `contextUsed = tokens.input + tokens.cache.read` is correct only if `input` EXCLUDES cached tokens (Anthropic semantics). For OpenAI, raw `prompt_tokens` historically INCLUDES cached tokens — if the upstream `LLM.Usage` isn't normalized per-provider, `used` double-counts cache reads, and the EVENTS.md "a token is counted in exactly one bucket" guarantee silently fails for those providers.
2. `reasoning: info.tokens.reasoning` emits `undefined` (dropped by JSON.stringify → missing field) if any provider's `StepFinishPart` doesn't populate `reasoning`.
Please confirm the upstream normalization covers every provider before relying on these metrics; otherwise guard with `(info.tokens.reasoning ?? 0)` and verify `input` excludes cache for OpenAI.
6. packages/cli/test/cli/usage-token-breakdown.test.ts:217 — Source-substring tests are brittle
   Detail: The "emit block shape" describe block reads `run.ts` as raw text and asserts on substrings (`"reasoning"`, `"cache"`, `"buildContextWindow"`, `"context"`). These assert on source *text*, not behavior: a rename/reformat breaks the test while behavior is unchanged (false negative), and a coincidental substring elsewhere in the 1500/500-char window could falsely pass (false positive). Prefer capturing the actual emitted event in a small integration test, or at least tighten the window to the exact emit object literal.

📋 Out-of-diff findings (6)

Sev	Location	Finding
⚪	EVENTS.md:91	Example ratio 0.049 ≠ emitted 0.04912
⚪	packages/cli/src/cli/cmd/run.ts:88	JSDoc hardcodes provider.ts:929 (line rot)
🟡	packages/cli/src/cli/cmd/run.ts:102	ratio unclamped but documented as 0–1
🟡	packages/cli/src/cli/cmd/run.ts:509	.catch(()=>null) swallows non-not-found errors
🟡	packages/cli/src/cli/cmd/run.ts:510	Verify upstream tokens shape for all providers
⚪	packages/cli/test/cli/usage-token-breakdown.test.ts:217	Source-substring tests are brittle

Reviewed 3 files · 0 inline · view all 6 findings ↗

---

aictrl · AI code review for fast-moving teams · aictrl.dev