Recognize _meta.replaces / _meta.collapsedCalls on tool-result ingest (#219)#226
Recognize _meta.replaces / _meta.collapsedCalls on tool-result ingest (#219)#226willwashburn merged 3 commits intomainfrom
Conversation
…#219) Replacement tools (e.g. relaywash) ship `_meta` annotations on their tool_result describing the counterfactual: `replaces` lists the built-in tools the call substituted for and `collapsedCalls` is the estimated count of vanilla calls collapsed into one. Burn now reads those annotations on Claude Code session ingest, persists them as per-call attribution on `ToolCall` and `ToolResultEventRecord`, and surfaces estimated tokens saved across the analyze and CLI layers. - reader: back-populate `replacedTools` / `collapsedCalls` from `tool_result._meta` onto matching `ToolCall` and `ToolResultEventRecord` records. - analyze: add `summarizeReplacementSavings()` / `estimateSavingsForToolCall()` with a static per-replaced-tool token cost lookup (option a from the issue; option b — averaging from the live ledger — is left as a future refinement). - cli: `burn summary` shows a top-line "estimated savings from replacement tools" notice and a `replacementSavings` JSON block when any annotated calls are present; `burn summary --by-tool` adds a `savedTokens` column and per-tool `savings` field in --json. Sessions without `_meta` annotations behave exactly as before.
There was a problem hiding this comment.
Devin Review found 1 potential issue.
🐛 1 issue in files not directly in the diff
🐛 Replacement savings always zero on default archive code path because replacedTools/collapsedCalls are not persisted in SQLite (packages/ledger/src/archive-query.ts:261-271)
The tool_calls archive table schema (packages/ledger/src/archive.ts:175-186) only stores tool_use_id, tool_name, target, args_hash, is_error — it has no columns for replacedTools or collapsedCalls. The archive insert at packages/ledger/src/archive.ts:785-795 writes only those columns, and the archive query at packages/ledger/src/archive-query.ts:261-271 reconstructs ToolCall without the new fields. Since burn summary defaults to the archive path via loadTurns() → queryAllFromArchive(), summarizeReplacementSavings(turns) will always see empty annotations and return zero savings. The CLI tests hide this because they set RELAYBURN_ARCHIVE=0 (line 1219), forcing the fallback ledger-walk path that preserves all TurnRecord fields from JSONL. In production, users will never see replacement savings in burn summary output despite their sessions carrying the _meta annotations.
View 5 additional findings in Devin Review.
Devin Review on #226 caught that the archive code path drops the new counterfactual annotations: `tool_calls` had no columns for them, the writer didn't insert them, and the reader rebuilt `ToolCall` without them. Since `burn summary` defaults to the archive path via `queryAllFromArchive()`, production users would never have seen any replacement-tool savings despite their sessions carrying `_meta` annotations — the CLI tests masked this by forcing `RELAYBURN_ARCHIVE=0` (fallback ledger walk). - Add `replaced_tools` (JSON) and `collapsed_calls` (INTEGER) columns to `tool_calls` via the existing additive-migration mechanism so reopened archives upgrade in place without a rebuild. - Insert + rehydrate the fields in the writer / `archive-query.ts`. - Tolerate malformed JSON on the read side rather than failing the whole archive query. - Cover with a `queryAllFromArchive()` round-trip test and a `burn summary` test that exercises the archive path (with `RELAYBURN_ARCHIVE` un-set), so this regression can't sneak back in.
…aRoJ9 # Conflicts: # CHANGELOG.md # packages/analyze/CHANGELOG.md # packages/cli/CHANGELOG.md # packages/ledger/CHANGELOG.md # packages/reader/CHANGELOG.md
Closes #219.
Summary
Replacement tools (e.g. relaywash, see AgentWorkforce/wash#1) ship a
_metafield on theirtool_resultdescribing the counterfactual:{ "_meta": { "replaces": ["Glob", "Grep", "Read"], "collapsedCalls": 9 } }Burn now ingests those annotations from Claude Code session logs, persists them as per-call attribution, and surfaces estimated tokens saved across the analyze and CLI layers.
Changes
@relayburn/readerToolCallandToolResultEventRecordgain optionalreplacedTools: string[]andcollapsedCalls: numberfields._meta.replaces/_meta.collapsedCallsfrom eachtool_resultblock (top-level_meta, or nested inside structuredcontent) and back-populates the matchingToolCallandToolResultEventRecord. Pattern mirrors howis_erroris back-propagated today.@relayburn/analyzereplacement-savingsmodule:DEFAULT_REPLACED_TOOL_TOKEN_COSTstatic lookup (Bash, Grep, Read, Edit, Write, Glob, LS, Task, WebFetch, WebSearch, …).estimateSavingsForToolCall(call)— per-call estimate.summarizeReplacementSavings(turns)— aggregate withbyToolbreakdown.@relayburn/cliburn summaryadds a top-lineestimated savings from replacement tools: ~N tokens across K calls (M collapsed vanilla calls)notice when any annotated calls are present, and areplacementSavingsblock in--json.burn summary --by-tooladds asavedTokenscolumn to the table and asavingsfield on each annotated row in--json.Backwards compatibility
_metaannotations behave exactly as before._metaparsing is tolerant — non-Claude sources, or Claude sessions with no replacement tools, see no behavior change.Test plan
pnpm run buildcleanpnpm run test— 903 tests pass (10 new: 1 reader, 6 analyze, 3 cli summary)tests/fixtures/claude/replacement-meta.jsonlexercises arelaywash__Searchcall collapsing 9 vanillaGlob/Grep/Readcalls, plus a vanillaReadcall without annotations to confirm both paths.replacementSavingsblock appears in default--json, is omitted when no annotations are present, and that--by-tool --jsonattaches asavingsfield only to annotated rows.https://claude.ai/code/session_01EK2PgJetMBDiPA4H67biXS
Generated by Claude Code