Measuring a Batch

Measuring a Batch

After each batch, compare two things: cost per change, and what the critic caught against what got past it.

Cost per change

workflow-cost.py reads the per-agent transcripts a Claude Code workflow leaves behind, sums tokens by model family, prices them, and prints cost per change against a baseline.

python3 workflow-cost.py <run-id> --prs 4 --baseline-per-pr 13

Point it at a run directory (.../subagents/workflows/wf_<id>) or a bare run id, which it looks up under every project in ~/.claude/projects. Several runs can be given at once.

Prices are list prices per million tokens in the PRICES table in lib/leanstats.py, by model version, with 1 hour cache writes counted at their own rate. Each message is priced on its own, because Haiku 5.5 charges a higher rate when one request's prompt is over 100,000 tokens; that rate is in LONG_PROMPT beside PRICES. Edit the table to what you pay. The script imports the table and the transcript reader from lib/leanstats.py, so copy both if you use it outside the plugin.

The 1.4.0 correction

Claude Code writes one transcript row per content block and repeats the message's usage on each. Until 1.4.0 workflow-cost.py summed every row, which overstated each of the 17 runs it was rechecked on by 65% to 304%. Its price table was also out of date: Fable was priced as Opus, and Opus and Sonnet at the prices of their version 4 models.

Counted once, at the list prices of 2026-10-02, the three baseline runs come to 225 dollars over 18 changes, 13 per change, and the first lean run to 17 dollars over 2 changes, 9 per change: 31% under, where the old figures said 61%. Any baseline you took with an older version is too high by the same kind of margin. Rerun it before you compare.

lean-stats

lean-stats covers what workflow-cost.py does not: agents started with the Agent tool, the main session, review outcomes, merged changes and hook fires. It keeps one SQLite file, ~/.lean/lean.db.

lean-stats collect --since 2026-10-01 owner/repo
lean-stats report --since 2026-10-01 [--markdown]
lean-stats import-reviews review-log.tsv

collect imports transcripts and merged pull requests. report prints cost per change by model and by role, the escalation rate, findings by where they were caught, critic misses, fix rounds and hook fires, each with the count behind it. import-reviews loads an existing review-log.tsv.

A line that says no data is a gap in what was recorded, not a zero.

An agent is counted by role only when it carries a label, lean:<role> <owner/repo>#<pr> (or @<branch> before the pull request exists), as its Agent description or as the first line of a workflow prompt. The roles are draft, spec, hunt, refute, escalate, search, gate and retro. The method and adversarial-review skills write the label. Without one the agent is counted as other, and the report's last line says what share of tokens that was.

LEAN_DB moves the file and LEAN_STATS=off stops every write. Nothing leaves the machine. No prompt text, code or command line is stored: tokens, model, role, repository and change only.

Measure cost, not tokens. On one comparable task the token count was the same on both models and the whole saving was in the rate.

With LEAN_MODELS=one the per-model split does not apply.

The review log

log-review.sh appends one line per review to review-log.tsv, or to the path in REVIEW_LOG.

log-review.sh <repo> <pr> <confirmed> <refuted> <blockers> "<categories>" "<caught elsewhere first>" [pre_push] [ci] [review] [escaped]

Column	Meaning
date	UTC date of the log line
repo, pr	The change
confirmed	Findings that survived refutation
refuted	Findings the refuter dropped
blockers	Confirmed findings of blocker severity
categories	Free text, the kinds of finding
caught_elsewhere_first	What CI or a review bot had already caught
pre_push	Findings caught before the push
ci	Findings caught by CI
review	Findings caught in review
escaped	Defects traced back to this change after it merged

The last four default to 0. escaped is filled in later: when a defect is traced back to a merged change, amend that row and do not add a new one. An older eight column log is widened in place on the next write.

The same row also goes into the lean-stats database. Logging the same pull request again replaces its row there and keeps the first review's date, which is how escaped gets filled in.

Reading it

Fewer review findings is not a goal. A change with no findings is either a clean change or a weak review, and the count cannot tell you which.

Claim	What it looks like in the log
Findings move earlier	pre_push rises as a share, and the total does not fall
The code is getting better	escaped falls
The review still works	review stays non-zero while escaped stays at zero
The review has gone soft	review falls and escaped rises

The last two rows are identical if you only count findings.

Batch signals

retro-signals.sh prints one TSV row per merged pull request. The retro starts from lean-stats report and uses this for the raw row per change.

REVIEW_LOG=review-log.tsv retro-signals.sh --since 2026-09-24 owner/repo [owner/repo ...]

Per pull request: commits pushed after it opened (fix rounds), failed check runs, days from open to merge, and later pull requests or issues that point back at it as the cause of a fix. With REVIEW_LOG set, it counts merged pull requests that have no log entry. It needs gh and jq, and writes only to stdout.

What to do with the numbers

• When the critic misses something, add a scripted check. Do not buy a more expensive critic.
• When a critic finds nothing on a tier for three batches, drop it from that tier.
• Feed the rest to the Retro.