Open
Conversation
Wire the `summarizeFidelity` / `hasMinimumFidelity` helpers from `@relayburn/analyze` (shipped in 0.14.0 by #41) into `burn compare` so the aggregate stops silently averaging `aggregate-only` / `cost-only` / `partial` turns with full-fidelity peers from the same model — the "looks more confident than it should" failure mode #41 called out. - Default minimum is `usage-only`. Records emitted before TurnRecord.fidelity existed (pre-#41 ledgers) still pass for backward compat. - `--fidelity <class>` overrides the floor; `--include-partial` is shorthand for `--fidelity partial` and includes every turn. Conflicting combinations exit 2 with a clear message. - TTY output adds an `excluded N turns below <class> fidelity (… aggregate- only, … cost-only, … partial)` coverage line whenever the gate dropped anything. Per-model totals render `—` instead of `$0.00` when a model survived the filter with zero turns. - JSON gains a top-level `fidelity` block (`{ minimum, excluded, summary }`) computed against the unfiltered slice so consumers can render their own coverage UI. - `runCompare` now accepts an optional `CompareDeps` for ingest/query/pricing injection (parallels `runLimits`), enabling deterministic CLI tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 26, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
summarizeFidelity/hasMinimumFidelityhelpers from@relayburn/analyze(shipped in 0.14.0 by #41, #76) intoburn compareso the aggregate stops silently averagingaggregate-only/cost-only/partialturns with full-fidelity peers from the same model — the "looks more confident than it should" failure mode Coverage and fidelity metadata: distinguish missing, zero, aggregate-only, and partial usage #41 called out.usage-only. Records emitted beforeTurnRecord.fidelityexisted (pre-Coverage and fidelity metadata: distinguish missing, zero, aggregate-only, and partial usage #41 ledgers) still pass for backward compat. New--fidelity <class>flag overrides the floor;--include-partialis shorthand for--fidelity partialand includes every turn.excluded N turns below <class> fidelity (… aggregate-only, … cost-only, … partial)coverage line whenever the gate dropped anything. Per-model totals render—instead of$0.00when a model survived the filter with zero turns.fidelityblock ({ minimum, excluded, summary }) computed against the unfiltered slice so consumers can render their own coverage UI.Implementation notes
runComparenow takes an optionalCompareDeps(ingestAll,queryAll,loadPricing), parallel torunLimits. The dispatch site incli.tsis unchanged — defaults pick up the production implementations. This is what enables the new in-process tests.FidelityClassordering inanalyze/src/fidelity.tsputspartialstrictly aboveaggregate-only/cost-only, so a literalhasMinimumFidelity(t.fidelity, 'partial')predicate would still drop those two buckets. The issue spec explicitly wants--fidelity partial(and--include-partial) to mean "no filtering at all", so that level is implemented as a runtime bypass rather than an order-table change. Documented in the inline comments next to the predicate andcomputeExcluded.Rebase consideration for PR #130
#130 (issue #88) is migrating
burn compareto read fromarchive.sqlite. This PR targets the in-memory path onmain. When #130 merges, the rebase needs to apply the same fidelity gating to the SQL-aggregate path — either filter thearchive.sqlitequery byattribution_fidelity(preferred), or hydrate enough rows to applyhasMinimumFidelityin TS the way this PR does. The exclusion summary, JSONfidelityblock, and "excluded N turns" coverage note all need to ride along through whichever path the rebase keeps. The--fidelity/--include-partialflag parsing inrunCompareis path-agnostic and should survive verbatim.Test plan
pnpm run buildpassespnpm run test:tspasses (525 tests, was 514)packages/cli/src/commands/compare.test.tscovers:aggregate-only/cost-only/partialfidelity === undefined--fidelity fullstrict mode--fidelity partial/--include-partial"include everything"--include-partial+ conflicting--fidelityexits 2--fidelityvalue exits 2fidelityblock shape (minimum,excluded,summary)—, not$0.00/0%Refs