fix: report agent usage per provider round, not once per turn - #4545
Conversation
buzz-agent emitted its `usage_update` notification in exactly one place: after `ctx.run()` returned. Until then a turn's token counters lived only in the prompt task's stack frame. A turn that was killed mid-flight therefore reported nothing at all — the provider had already billed every round it completed, and no consumer ever saw any of it. That is not a corner case for anything that terminates a turn on a clock. Measured on a long-horizon benchmark run that relaunches its agent between phases, 97% of the recorded usage rows came back all zeros, and the run's recorded cost was $79.23 against the $485 the provider's own ledger billed for the same model over the same window — a 5.9x undercount, concentrated on the seat being relaunched. Nothing was truncated after the fact; the tokens were never captured in the first place. The run loop now emits a session-cumulative `usage_update` after every usage-bearing provider response, so an interrupted turn has reported everything but its single in-flight request. Emitting more than once per turn is already part of the contract — buzz-acp's UsageTracker advances its committed baseline only at publish time, and goose behaves the same way, which is why the tracker was written to tolerate it. The turn-start session baseline is snapshotted into RunCtx so the mid-turn figure stays session-cumulative rather than turn-local; a turn-local number would be discarded by a high-water-mark consumer and lose the turn entirely. It is taken by value rather than as a handle because the loop reports once per round, and taking the sessions lock on each of those would serialise concurrent sessions behind one another's provider round-trips. Both call sites now build the payload through one helper so their wire shapes cannot drift — a divergence there would present as tokens silently vanishing, which is the failure this reporting exists to prevent. A SIGTERM handler was the obvious shape and does not work: at signal time the counters are not anywhere a handler could reach. Making usage durable during the turn is what fixes it. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
7344f68 to
ec9de1d
Compare
wpfleger96
left a comment
There was a problem hiding this comment.
I checked this against both the token-usage UI contract and NIP-AM. The per-round ACP notifications remain ephemeral telemetry: UsageTracker accepts multiple updates during one turn, keeps the prior published baseline frozen, lets the latest update win, and only advances turnSeq when take() publishes. This still produces exactly one durable kind-44200 usage event per completed turn.
The session-cumulative payload is the right shape for interrupted turns and preserves provenance: model identity, cached-input tokens, and genuine provider-reported totals remain intact rather than being reconstructed. The final end-of-turn notification is compatible with the intermediate updates.
Minimalism, elegance, and correctness all clear the gate: the baseline snapshot avoids per-round session locking, payload construction is centralized, and the tests cover multiple provider rounds plus prior-turn accumulation.
Non-blocking integration caveat: #4553 addressed settling this partial usage before the benchmark harness kills an agent on timeout, but it was closed without merge and I found no open replacement. That timeout-side fix still needs to land for LHTB accounting to benefit from the newly observable partial usage; it is not a defect in this PR.
…ed-unread * origin/main: (44 commits) chore(release): release Buzz Desktop version 0.5.4 (#4562) test(mobile): assert follow boundary semantics (#4559) docs(release): align desktop handoff instructions (#3988) fix: report agent usage per provider round, not once per turn (#4545) fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382) feat(desktop): improve channel template discovery (#4549) fix(desktop): save key backups to authorized path (#4022) Add channel activity hover menu (#3935) feat(desktop): show saved Run on settings when editing an agent (#4539) fix(desktop): disambiguate provider API key labels and annotate mint key (#4406) fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140) fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580) Polish mobile composer and messaging UI (#3918) ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524) fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411) fix(mobile): recover and pace live subscriptions (#3053) feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395) fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392) fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778) feat(k8s): Kubernetes backend plugin + desktop deploy path (#4289) ... Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> * origin/main: (40 commits) fix(mobile): recover stale relay sessions (#4372) chore(release): release Buzz Desktop version 0.5.4 (#4562) test(mobile): assert follow boundary semantics (#4559) docs(release): align desktop handoff instructions (#3988) fix: report agent usage per provider round, not once per turn (#4545) fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382) feat(desktop): improve channel template discovery (#4549) fix(desktop): save key backups to authorized path (#4022) Add channel activity hover menu (#3935) feat(desktop): show saved Run on settings when editing an agent (#4539) fix(desktop): disambiguate provider API key labels and annotate mint key (#4406) fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140) fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580) Polish mobile composer and messaging UI (#3918) ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524) fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411) fix(mobile): recover and pace live subscriptions (#3053) feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395) fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392) fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778) ...
Syncs Cargo.lock and ci.yml updates from origin/main via the umbrella, which resolves the RUSTSEC-2026-0225..0229 nostr advisory failures in the Security gate. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> * commit 'ad9ae1dbb0c2d9b6839c1e18d6fd6a78238b61fb': (40 commits) fix(mobile): recover stale relay sessions (#4372) chore(release): release Buzz Desktop version 0.5.4 (#4562) test(mobile): assert follow boundary semantics (#4559) docs(release): align desktop handoff instructions (#3988) fix: report agent usage per provider round, not once per turn (#4545) fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382) feat(desktop): improve channel template discovery (#4549) fix(desktop): save key backups to authorized path (#4022) Add channel activity hover menu (#3935) feat(desktop): show saved Run on settings when editing an agent (#4539) fix(desktop): disambiguate provider API key labels and annotate mint key (#4406) fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140) fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580) Polish mobile composer and messaging UI (#3918) ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524) fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411) fix(mobile): recover and pace live subscriptions (#3053) feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395) fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392) fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778) ...
The bug
buzz-agent emitted its
usage_updatenotification in exactly one place: afterctx.run()returned. Until that moment a turn's token counters lived only in the prompt task's stack frame. A turn killed mid-flight reported nothing at all — the provider had already billed every round it completed, and no consumer ever saw any of it.That is not a corner case for anything that ends a turn on a clock. It is the normal case for a long-horizon benchmark run that relaunches its agent between phases.
How big
Measured against a provider's own billing ledger over one run's window:
97% of that run's usage rows came back all zeros, against 1–4% for comparable runs that never relaunch. In one 450-phase trial exactly 7 phases recorded any usage — and each of those carries 177k–437k input tokens, a whole session's worth landing in the one phase that happened to end gracefully.
Worth being precise about what was not wrong, since both were plausible and both were checked:
The fix
The run loop now emits a session-cumulative
usage_updateafter every usage-bearing provider response, so an interrupted turn has reported everything but its single in-flight request.UsageTrackeradvances its committed baseline only at publish time, and goose behaves the same way — which is why the tracker was written to tolerate it.RunCtxso the mid-turn figure stays session-cumulative. A turn-local number would be discarded by a high-water-mark consumer and lose the turn entirely; there is a test for exactly that.busy, so it cannot go stale.wire::usage_update_payloadfor both call sites, so the mid-turn and end-of-turn shapes cannot drift. A drift there would present as tokens silently vanishing, which is the failure this reporting exists to prevent.Why not a SIGTERM handler
That was the obvious shape and it does not work. At signal time the counters are not sitting anywhere a handler could reach — they are in the turn's stack frame, and the value the handler would need has not been folded into the session yet. Making usage durable during the turn is what actually fixes it; once it is, a handler adds nothing beyond the in-flight request, whose cost is unknown until its response lands.
Tests
usage_is_reported_after_each_round_not_only_at_turn_end— two rounds; asserts the first notification carries round 1's counts alone, proving it went out before round 2 returned.mid_turn_usage_includes_earlier_turns— a mid-turn report must be session-cumulative, not turn-local.buzz-agent 18/18 on the
fake_llmsuite, 382 unit.cargo fmt/clippy/cargo check --workspace --all-targetsclean.Scope
Agent-side only, against
main. The matching harness change — settling usage on the timeout path, which was skipped on the reasoning that an incomplete turn has nothing to flush — is #4553, against the benchmark branch, since that harness does not exist onmain.🤖 Generated with Claude Code