Skip to content

fix: report agent usage per provider round, not once per turn - #4545

Merged
atishpatel merged 1 commit into
mainfrom
benchmark/flush-usage-log-per-phase
Aug 3, 2026
Merged

fix: report agent usage per provider round, not once per turn#4545
atishpatel merged 1 commit into
mainfrom
benchmark/flush-usage-log-per-phase

Conversation

@atishpatel

@atishpatel atishpatel commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The bug

buzz-agent emitted its usage_update notification in exactly one place: after ctx.run() returned. Until that moment a turn's token counters lived only in the prompt task's stack frame. A turn killed mid-flight reported nothing at all — the provider had already billed every round it completed, and no consumer ever saw any of it.

That is not a corner case for anything that ends a turn on a clock. It is the normal case for a long-horizon benchmark run that relaunches its agent between phases.

How big

Measured against a provider's own billing ledger over one run's window:

provider ledger what we recorded
the relaunched lead seat $485 / 348M tok $98.99 / 90.3M tok
the two seats that were not relaunched $29.90 / 856M $25.81 / 765M — reconciles

97% of that run's usage rows came back all zeros, against 1–4% for comparable runs that never relaunch. In one 450-phase trial exactly 7 phases recorded any usage — and each of those carries 177k–437k input tokens, a whole session's worth landing in the one phase that happened to end gracefully.

Worth being precise about what was not wrong, since both were plausible and both were checked:

  • Not pricing. The rates were verified against the provider's endpoints API and match what we charge.
  • Not a truncation bug. The usage files were intact and internally consistent. The tokens were never captured in the first place.

The fix

The run loop now emits a session-cumulative usage_update after every usage-bearing provider response, so an interrupted turn has reported everything but its single in-flight request.

  • Emitting more than once per turn is already part of the contract. buzz-acp's UsageTracker advances its committed baseline only at publish time, and goose behaves the same way — which is why the tracker was written to tolerate it.
  • The turn-start session baseline is snapshotted into RunCtx so the mid-turn figure stays session-cumulative. A turn-local number would be discarded by a high-water-mark consumer and lose the turn entirely; there is a test for exactly that.
  • Snapshot by value, not a session handle. The loop reports once per round, and taking the sessions lock on each would serialise concurrent sessions behind one another's provider round-trips. Nothing else advances those counters while the turn holds busy, so it cannot go stale.
  • One shared wire::usage_update_payload for both call sites, so the mid-turn and end-of-turn shapes cannot drift. A drift there would present as tokens silently vanishing, which is the failure this reporting exists to prevent.

Why not a SIGTERM handler

That was the obvious shape and it does not work. At signal time the counters are not sitting anywhere a handler could reach — they are in the turn's stack frame, and the value the handler would need has not been folded into the session yet. Making usage durable during the turn is what actually fixes it; once it is, a handler adds nothing beyond the in-flight request, whose cost is unknown until its response lands.

Tests

  • usage_is_reported_after_each_round_not_only_at_turn_end — two rounds; asserts the first notification carries round 1's counts alone, proving it went out before round 2 returned.
  • mid_turn_usage_includes_earlier_turns — a mid-turn report must be session-cumulative, not turn-local.

buzz-agent 18/18 on the fake_llm suite, 382 unit. cargo fmt / clippy / cargo check --workspace --all-targets clean.

Scope

Agent-side only, against main. The matching harness change — settling usage on the timeout path, which was skipped on the reasoning that an incomplete turn has nothing to flush — is #4553, against the benchmark branch, since that harness does not exist on main.

🤖 Generated with Claude Code

@atishpatel
atishpatel requested a review from a team as a code owner August 3, 2026 15:35
buzz-agent emitted its `usage_update` notification in exactly one place:
after `ctx.run()` returned. Until then a turn's token counters lived only
in the prompt task's stack frame. A turn that was killed mid-flight
therefore reported nothing at all — the provider had already billed every
round it completed, and no consumer ever saw any of it.

That is not a corner case for anything that terminates a turn on a clock.
Measured on a long-horizon benchmark run that relaunches its agent between
phases, 97% of the recorded usage rows came back all zeros, and the run's
recorded cost was $79.23 against the $485 the provider's own ledger billed
for the same model over the same window — a 5.9x undercount, concentrated
on the seat being relaunched. Nothing was truncated after the fact; the
tokens were never captured in the first place.

The run loop now emits a session-cumulative `usage_update` after every
usage-bearing provider response, so an interrupted turn has reported
everything but its single in-flight request. Emitting more than once per
turn is already part of the contract — buzz-acp's UsageTracker advances its
committed baseline only at publish time, and goose behaves the same way,
which is why the tracker was written to tolerate it.

The turn-start session baseline is snapshotted into RunCtx so the mid-turn
figure stays session-cumulative rather than turn-local; a turn-local number
would be discarded by a high-water-mark consumer and lose the turn
entirely. It is taken by value rather than as a handle because the loop
reports once per round, and taking the sessions lock on each of those would
serialise concurrent sessions behind one another's provider round-trips.
Both call sites now build the payload through one helper so their wire
shapes cannot drift — a divergence there would present as tokens silently
vanishing, which is the failure this reporting exists to prevent.

A SIGTERM handler was the obvious shape and does not work: at signal time
the counters are not anywhere a handler could reach. Making usage durable
during the turn is what fixes it.

Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
@atishpatel
atishpatel force-pushed the benchmark/flush-usage-log-per-phase branch from 7344f68 to ec9de1d Compare August 3, 2026 16:01
@atishpatel
atishpatel changed the base branch from benchmark/harness-accounting-and-solo to main August 3, 2026 16:01

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this against both the token-usage UI contract and NIP-AM. The per-round ACP notifications remain ephemeral telemetry: UsageTracker accepts multiple updates during one turn, keeps the prior published baseline frozen, lets the latest update win, and only advances turnSeq when take() publishes. This still produces exactly one durable kind-44200 usage event per completed turn.

The session-cumulative payload is the right shape for interrupted turns and preserves provenance: model identity, cached-input tokens, and genuine provider-reported totals remain intact rather than being reconstructed. The final end-of-turn notification is compatible with the intermediate updates.

Minimalism, elegance, and correctness all clear the gate: the baseline snapshot avoids per-round session locking, payload construction is centralized, and the tests cover multiple provider rounds plus prior-turn accumulation.

Non-blocking integration caveat: #4553 addressed settling this partial usage before the benchmark harness kills an agent on timeout, but it was closed without merge and I found no open replacement. That timeout-side fix still needs to land for LHTB accounting to benefit from the newly observable partial usage; it is not a defect in this PR.

@atishpatel
atishpatel merged commit 09c86c5 into main Aug 3, 2026
40 of 46 checks passed
@atishpatel
atishpatel deleted the benchmark/flush-usage-log-per-phase branch August 3, 2026 16:42
wpfleger96 pushed a commit that referenced this pull request Aug 3, 2026
…ed-unread

* origin/main: (44 commits)
  chore(release): release Buzz Desktop version 0.5.4 (#4562)
  test(mobile): assert follow boundary semantics (#4559)
  docs(release): align desktop handoff instructions (#3988)
  fix: report agent usage per provider round, not once per turn (#4545)
  fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382)
  feat(desktop): improve channel template discovery (#4549)
  fix(desktop): save key backups to authorized path (#4022)
  Add channel activity hover menu (#3935)
  feat(desktop): show saved Run on settings when editing an agent (#4539)
  fix(desktop): disambiguate provider API key labels and annotate mint key (#4406)
  fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140)
  fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580)
  Polish mobile composer and messaging UI (#3918)
  ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524)
  fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411)
  fix(mobile): recover and pace live subscriptions (#3053)
  feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395)
  fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392)
  fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778)
  feat(k8s): Kubernetes backend plugin + desktop deploy path (#4289)
  ...

Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 added a commit that referenced this pull request Aug 3, 2026
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

* origin/main: (40 commits)
  fix(mobile): recover stale relay sessions (#4372)
  chore(release): release Buzz Desktop version 0.5.4 (#4562)
  test(mobile): assert follow boundary semantics (#4559)
  docs(release): align desktop handoff instructions (#3988)
  fix: report agent usage per provider round, not once per turn (#4545)
  fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382)
  feat(desktop): improve channel template discovery (#4549)
  fix(desktop): save key backups to authorized path (#4022)
  Add channel activity hover menu (#3935)
  feat(desktop): show saved Run on settings when editing an agent (#4539)
  fix(desktop): disambiguate provider API key labels and annotate mint key (#4406)
  fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140)
  fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580)
  Polish mobile composer and messaging UI (#3918)
  ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524)
  fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411)
  fix(mobile): recover and pace live subscriptions (#3053)
  feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395)
  fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392)
  fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778)
  ...
wpfleger96 added a commit that referenced this pull request Aug 3, 2026
Syncs Cargo.lock and ci.yml updates from origin/main via the umbrella, which
resolves the RUSTSEC-2026-0225..0229 nostr advisory failures in the Security gate.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

* commit 'ad9ae1dbb0c2d9b6839c1e18d6fd6a78238b61fb': (40 commits)
  fix(mobile): recover stale relay sessions (#4372)
  chore(release): release Buzz Desktop version 0.5.4 (#4562)
  test(mobile): assert follow boundary semantics (#4559)
  docs(release): align desktop handoff instructions (#3988)
  fix: report agent usage per provider round, not once per turn (#4545)
  fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382)
  feat(desktop): improve channel template discovery (#4549)
  fix(desktop): save key backups to authorized path (#4022)
  Add channel activity hover menu (#3935)
  feat(desktop): show saved Run on settings when editing an agent (#4539)
  fix(desktop): disambiguate provider API key labels and annotate mint key (#4406)
  fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140)
  fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580)
  Polish mobile composer and messaging UI (#3918)
  ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524)
  fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411)
  fix(mobile): recover and pace live subscriptions (#3053)
  feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395)
  fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392)
  fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants