v0.2.34: fix: report swept patterns per run, and add a deterministic sweep replay
Two changes: a bug in a number that was about to be published, and the artifact that publishes it.
1. The swept-pattern accumulator leaked across runs
A cycle receipt on #67 printed Swept 2 patterns. on a run where no --swept-pattern was passed at all.
The convergence block unioned the live accumulator with the swept signatures of every prior recorded run of the PR, then wrote that union back into the new record. --record-run did clear the live accumulator — but pattern_history_for_pr read the count straight back out of runs.jsonl, and each new record re-persisted the union. One run that swept two patterns made every later run of that PR report Swept 2 patterns forever, and the number could never fall again.
Reproduced before fixing:
--- cycle 1: agent passes --swept-pattern twice ---
receipt line : Convergence: 1 distinct patterns this cycle, 0 recurred. Swept 2 patterns.
--record-run: appended record, cleared live accumulator
--- cycle 2: fresh run, NO --swept-pattern passed ---
live accumulator : [] <- correctly empty
swept_sigs used : ['64250ca5', 'exception-wrap'] <- leaked back in from runs.jsonl
receipt line : Convergence: 1 distinct patterns this cycle, 0 recurred. Swept 2 patterns.
The two sets answer different questions and are now kept apart in swept_pattern_sets():
| meaning | drives | |
|---|---|---|
this_run |
what --swept-pattern supplied on this run |
the receipt count, and what gets persisted |
ever |
plus signatures from prior runs of the PR | recurrence detection only |
Persisting this_run rather than ever is what stops the compounding — pattern_history_for_pr already unions across records, so writing the union back into each record made it permanent.
Recurrence detection is unaffected: a pattern swept in an earlier cycle that reappears now is exactly what the advisory is for, and there is a test pinning that it still fires.
The lifecycle decision used to be inline in main(), reachable only through a network fetch — which is why nothing caught it. It is now a named helper with three regression tests, verified red before the fix and green after:
=== RED without fix ===
FAILED tests/test_fetch_gemini_threads.py::TestSweptPatternLifecycle::test_a_run_that_swept_nothing_reports_zero
FAILED tests/test_fetch_gemini_threads.py::TestSweptPatternLifecycle::test_history_still_drives_recurrence_detection
FAILED tests/test_fetch_gemini_threads.py::TestSweptPatternLifecycle::test_persisting_the_union_would_compound_across_records
3 failed
=== fix restored ===
3 passed
2. evals/replay — deterministic sweep evidence
A live-reviewer demo is not reliable evidence. Sourcery returned two findings against b391162 and one finding against 67551bd, which is the same file contents re-pushed after a rebase. Retrying until a run happens to show the sweep would mean publishing the luckiest sample.
The sweep is deterministic, so this proves it deterministically over both captured payloads, and states the reviewer variance plainly rather than hiding it:
- run 1 — two findings worded differently → two prose clusters → one shape-merged cluster of two sites → sweep reports 3 unflagged siblings
- run 2 — one finding → one cluster of one site → correctly no sweep
$ python3 evals/replay/replay.py
-- clustered by finding prose alone --
Patterns (2):
[unknown] exception-wrap — 1 site (sig: exception-wrap)
loaders/profiles.py:17
[unknown] **suggestion (bug_risk):** add type and error checks around … — 1 site (sig: 64250ca5)
loaders/bundle.py:13
-- clustered by prose and code shape (shipping behaviour) --
Patterns (1):
[unknown] **suggestion:** defensively handle malformed or non-object j… — 2 sites (sig: shape:802659ea)
loaders/profiles.py:17, loaders/bundle.py:13
-- sweep --
[sweep] **suggestion:** defensively handle malformed or non-object j…
flagged: loaders/profiles.py:17, loaders/bundle.py:13
shared: data get {})
siblings: 3 unflagged site(s) match the same shape
+ loaders/bundle.py:25 return data.get("overrides", {})
+ loaders/profiles.py:23 return data.get("reviewers", {})
+ loaders/profiles.py:29 return data.get("defaults", {})
Fixtures are real captured data — bodies, paths, line anchors and author login copied verbatim from the recorded comment IDs (3721865840, 3721865850, 3731481473), with the reviewed loaders/ source vendored beside them so the replay does not depend on the demo branch surviving. replay.py takes no arguments; verified zero non-stdlib imports across the whole chain and no network or subprocess surface in it. Output is pinned by a committed golden file.
Verification
pytest -q→ 789 passed (782 + 7 replay)ruff check plugins/ tests/ evals/→ clean
🤖 Generated with Claude Code
https://claude.ai/code/session_014f8SDmEgdjSHo5iw4a3ykm
Summary by Sourcery
Ensure swept-pattern reporting reflects only the current run while adding a deterministic replay for sibling-sweep behaviour over captured reviewer payloads.
Bug Fixes:
- Prevent swept-pattern counts from accumulating across runs of the same PR so that receipts report only patterns swept in the current run.
Enhancements:
- Introduce a helper to separate current-run swept pattern signatures from historical ones and use it to drive recurrence detection and persistence correctly.
- Add a deterministic replay script that runs the real clustering and sibling-sweep logic over captured reviewer payloads and vendored source without external dependencies.
Documentation:
- Document the deterministic sweep replay fixtures, behaviour, and rationale in a new README under evals/replay/.
Tests:
- Add regression tests covering the swept-pattern lifecycle to ensure counts reset per run while still leveraging history for recurrence detection.
- Add tests that pin the deterministic replay output to a golden file, enforce determinism and portability, and verify fixture provenance and vendored source contents.