Skip to content

Recognize _meta.replaces / _meta.collapsedCalls on tool-result ingest (#219)#226

Merged
willwashburn merged 3 commits intomainfrom
claude/fix-issue-219-aRoJ9
May 2, 2026
Merged

Recognize _meta.replaces / _meta.collapsedCalls on tool-result ingest (#219)#226
willwashburn merged 3 commits intomainfrom
claude/fix-issue-219-aRoJ9

Conversation

@willwashburn
Copy link
Copy Markdown
Member

Closes #219.

Summary

Replacement tools (e.g. relaywash, see AgentWorkforce/wash#1) ship a _meta field on their tool_result describing the counterfactual:

{ "_meta": { "replaces": ["Glob", "Grep", "Read"], "collapsedCalls": 9 } }

Burn now ingests those annotations from Claude Code session logs, persists them as per-call attribution, and surfaces estimated tokens saved across the analyze and CLI layers.

Changes

@relayburn/reader

  • ToolCall and ToolResultEventRecord gain optional replacedTools: string[] and collapsedCalls: number fields.
  • The Claude reader collects _meta.replaces / _meta.collapsedCalls from each tool_result block (top-level _meta, or nested inside structured content) and back-populates the matching ToolCall and ToolResultEventRecord. Pattern mirrors how is_error is back-propagated today.

@relayburn/analyze

  • New replacement-savings module:
    • DEFAULT_REPLACED_TOOL_TOKEN_COST static lookup (Bash, Grep, Read, Edit, Write, Glob, LS, Task, WebFetch, WebSearch, …).
    • estimateSavingsForToolCall(call) — per-call estimate.
    • summarizeReplacementSavings(turns) — aggregate with byTool breakdown.
  • Implements option (a) from the issue (static lookup, cheap and approximate). Option (b) — averaging from the live ledger — is left as a future refinement once we have enough samples.

@relayburn/cli

  • burn summary adds a top-line estimated savings from replacement tools: ~N tokens across K calls (M collapsed vanilla calls) notice when any annotated calls are present, and a replacementSavings block in --json.
  • burn summary --by-tool adds a savedTokens column to the table and a savings field on each annotated row in --json.

Backwards compatibility

  • All new fields are optional. Sessions without _meta annotations behave exactly as before.
  • _meta parsing is tolerant — non-Claude sources, or Claude sessions with no replacement tools, see no behavior change.

Test plan

  • pnpm run build clean
  • pnpm run test — 903 tests pass (10 new: 1 reader, 6 analyze, 3 cli summary)
  • New fixture tests/fixtures/claude/replacement-meta.jsonl exercises a relaywash__Search call collapsing 9 vanilla Glob/Grep/Read calls, plus a vanilla Read call without annotations to confirm both paths.
  • CLI summary integration tests verify the replacementSavings block appears in default --json, is omitted when no annotations are present, and that --by-tool --json attaches a savings field only to annotated rows.

https://claude.ai/code/session_01EK2PgJetMBDiPA4H67biXS


Generated by Claude Code

…#219)

Replacement tools (e.g. relaywash) ship `_meta` annotations on their
tool_result describing the counterfactual: `replaces` lists the built-in
tools the call substituted for and `collapsedCalls` is the estimated
count of vanilla calls collapsed into one. Burn now reads those
annotations on Claude Code session ingest, persists them as per-call
attribution on `ToolCall` and `ToolResultEventRecord`, and surfaces
estimated tokens saved across the analyze and CLI layers.

- reader: back-populate `replacedTools` / `collapsedCalls` from
  `tool_result._meta` onto matching `ToolCall` and
  `ToolResultEventRecord` records.
- analyze: add `summarizeReplacementSavings()` /
  `estimateSavingsForToolCall()` with a static per-replaced-tool token
  cost lookup (option a from the issue; option b — averaging from the
  live ledger — is left as a future refinement).
- cli: `burn summary` shows a top-line "estimated savings from
  replacement tools" notice and a `replacementSavings` JSON block when
  any annotated calls are present; `burn summary --by-tool` adds a
  `savedTokens` column and per-tool `savings` field in --json.

Sessions without `_meta` annotations behave exactly as before.
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

🐛 1 issue in files not directly in the diff

🐛 Replacement savings always zero on default archive code path because replacedTools/collapsedCalls are not persisted in SQLite (packages/ledger/src/archive-query.ts:261-271)

The tool_calls archive table schema (packages/ledger/src/archive.ts:175-186) only stores tool_use_id, tool_name, target, args_hash, is_error — it has no columns for replacedTools or collapsedCalls. The archive insert at packages/ledger/src/archive.ts:785-795 writes only those columns, and the archive query at packages/ledger/src/archive-query.ts:261-271 reconstructs ToolCall without the new fields. Since burn summary defaults to the archive path via loadTurns()queryAllFromArchive(), summarizeReplacementSavings(turns) will always see empty annotations and return zero savings. The CLI tests hide this because they set RELAYBURN_ARCHIVE=0 (line 1219), forcing the fallback ledger-walk path that preserves all TurnRecord fields from JSONL. In production, users will never see replacement savings in burn summary output despite their sessions carrying the _meta annotations.

View 5 additional findings in Devin Review.

Open in Devin Review

claude and others added 2 commits May 2, 2026 12:54
Devin Review on #226 caught that the archive code path drops the new
counterfactual annotations: `tool_calls` had no columns for them, the
writer didn't insert them, and the reader rebuilt `ToolCall` without
them. Since `burn summary` defaults to the archive path via
`queryAllFromArchive()`, production users would never have seen any
replacement-tool savings despite their sessions carrying `_meta`
annotations — the CLI tests masked this by forcing
`RELAYBURN_ARCHIVE=0` (fallback ledger walk).

- Add `replaced_tools` (JSON) and `collapsed_calls` (INTEGER) columns to
  `tool_calls` via the existing additive-migration mechanism so reopened
  archives upgrade in place without a rebuild.
- Insert + rehydrate the fields in the writer / `archive-query.ts`.
- Tolerate malformed JSON on the read side rather than failing the whole
  archive query.
- Cover with a `queryAllFromArchive()` round-trip test and a
  `burn summary` test that exercises the archive path (with
  `RELAYBURN_ARCHIVE` un-set), so this regression can't sneak back in.
…aRoJ9

# Conflicts:
#	CHANGELOG.md
#	packages/analyze/CHANGELOG.md
#	packages/cli/CHANGELOG.md
#	packages/ledger/CHANGELOG.md
#	packages/reader/CHANGELOG.md
@willwashburn willwashburn merged commit 8782bd5 into main May 2, 2026
2 checks passed
@willwashburn willwashburn deleted the claude/fix-issue-219-aRoJ9 branch May 2, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recognize _meta.replaces / _meta.collapsedCalls annotations on tool-result ingest

2 participants