Close out Cut B: pin the walked candidates and source the last cell (#456)

[pengfei-threemoonslab]

Summary

Sourcing for the pre_1_0 corpus is finished. The strata inventory goes from 59 of 60 slots with a candidate (three of them unpinned) to 60 of 60 pinned, so nothing in it needs mining before the Cut C calibration round. One sweep — 2026-W36-closeout, 45 rows over three repositories — does all of it.

• The last gap. langchain_crewai × insufficient_evidence is claimed by bytedance/deer-flow#4868. Session B's lead was to stop mining langchain-mcp-adapters and find an application that calls MultiServerMCPClient / load_mcp_tools at agent construction; deer-flow is one, and the claimed PR adds per-user credential injection for shared MCP servers, so which credential a tool call carries is chosen at run time from the caller's identity and a $ENV_VAR map. Neither the servers, the tools, nor the credentials are in the tree.
• Three walk candidates, pinned. github-mcp-server#3020, #3076 and grafana/mcp-grafana#1080 were carried from adoption walks with abbreviated or absent SHAs, and the inventory said they had to be pinned before Cut C. Each was mined by number, which resolves both ends under the sweep convention and puts the pins where a guard can re-read them.
• The guard that section names did not exist. test_a_pinned_external_candidate_matches_the_sweep_that_recorded_it was added in Cut A and deleted by Cut A's own review commit (958a9073) — the same commit that added the prose citing it by name. Restored and generalized: every external pin is re-read from the sweep that resolved it, a pin no sweep corroborates is refused, a subject a sweep did resolve may not sit unpinned, and two recordings of one subject that disagree fail whether they sit in two sweeps or twice in one file.

Things a reviewer should know

• #3076 is why re-reading pins matters. Its walk note recorded head 5ea9a0e8…; that is refs/pull/3076/head, and after a squash merge it is not an ancestor of the default branch at all. Resolving the abbreviation "correctly" would have pinned an object no clone reaches. The merge commit is 8ec62491…, and its first parent is the bfb59bb7… the same note recorded — half the pin was right, which is the dangerous half.
• A Cut B finding is corrected, not repeated. The Cut B sweep concluded that openai-agents-python and the other SDK repositories init_skip because "the example trees are no longer picked up by cold-start init". Reproduced at the recorded pin, the actual result is refused_unresolved_scope: the repository holds more than one self-contained project that defines agents, and one manifest describes one agent surface. The refusal is the monorepo behavior working. What the miner cannot do is act on it — _run_init reads only "did a manifest appear", so a refusal that names the right next step is indistinguishable in the CSV from a crash.
• This matters for Cut D, not just for the CSV. 24 of the 40 deer-flow rows are init_skip for that reason; pointed at backend/packages/harness, init --write writes a manifest and the scan reaches a decision. Nine slots claimed before this cut are in the same position. A receipt is rooted at a project, not at a clone, and the inventory now says so.
• --limit 40 is not a time window. deer-flow's latest forty merged PRs span six days; crewAI-examples' forty reach back eighteen months. The claimed candidate is five days older than the window's oldest PR, which is why it took --pr rather than a wider sweep — and why a busy repository's silence under --limit 40 means very little.
• The three pinned walk candidates keep diff_substance. The walks are what targeted their cells; being swept afterwards adds benchmark_scored to their exposure and changes nothing about what aimed them. All three were already tuning_only through maintainer_walk, so no cell's holdout margin moved.
• Filling the last gap made a tripwire vacuous, by design. test_a_gap_that_names_a_pull_request_plans_the_origin_that_pr_can_supply ends with assert checked, "no gap names a pull request any more; drop this guard or restore one". It is re-pointed at the Reserve table, which states an origin beside a state exactly the way a gap did and is the pool a relabel draws on — 33 claims checked, and a reserve origin a state cannot supply now fails.
• The label on #4868 is one session's cell-targeting label, from the diff, and not adjudicated — like every other sourcing label under benchmark/miner/results/. Corpus labels come only from the Amendment 1 raters. The register records the counter-reading with it: the feature's default is fail-closed, so a rater may weigh the tightening rather than the unenumerable surface.

Review of this branch, and what it changed

Round 2 (8ff68ef5) — five findings, all addressed:

• [P2] Every results CSV is pin authority, but only *-mined runs were integrity-checked. _swept_pins reads every committed CSV, while test_miner_corpus.py validated well-formedness, CSV/JSONL parity and LF endings over *-mined.jsonl alone — so a half-regenerated -cutb, -closeout, -reeval or constructed pair could leave stale pins certifying the inventory. The three integrity guards now run over every committed run; only the KPI aggregation stays on *-mined. Two guards keep the split honest: the enumerations may not converge, and a results CSV with no JSONL sibling fails.
• [P2] The perturbation sweep was manual, so it protected nothing. Both parsers now take their input and the row-versus-sweep rule moved into _pin_complaints, so fixtures run through the same code the inventory does. The perturbations are committed — a contradictory duplicate inside one sweep in both row orders, an uncorroborated pin, a resolved subject left unpinned, an open PR reserved with a real origin, a misspelled reserve origin — and each was confirmed by reverting the implementation, not by construction.
• [P3 ×3] The #3076 row's note still carried a completed instruction; the run-table row still counted three candidates as three servers; and the inventory carried two current statuses (the opening paragraph, the origin-floor reading and the order-of-work line all still described sourcing as future work).

Round 1 (89cc708b) — two of the guards this PR adds failed open on the shape they exist for:

• _swept_pins keyed recordings by sweep file, so one file holding a subject twice with different SHAs kept only the last row — a contradictory duplicate ordered before the good row read as agreement. Confirmed by replaying the pre-fix keying against that perturbation. Now keyed by the pins, with the sweeps that recorded them as the value.
• _reserve_claims skipped every reserve row whose State was outside STATE_ORIGINS, which includes open — so adk-samples#1745, the candidate that taught this project that an open PR is not history, could have been reserved as real_history and passed. A row that states an origin must now state a state that can supply one.

Two prose claims were also overstated and corrected (86227ccb): "the three walked MCP servers" counted three candidates as three servers, and the deer-flow window figure said six weeks where the truth is five days.

Type

• Check or risk-model change
• Input adapter change
• CLI or GitHub Action behavior
• Report, schema, or SARIF output
• Benchmark corpus and documentation only — nothing under src/ changes

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest. All checks pass on the head commit.

Additional local checks run:

• pytest -n auto -m "not perf" on the full suite: exit 0; ruff check .: clean; compileall -q src tests: clean.
• The perturbations are committed as regression tests rather than run by hand, and each was verified by reverting the implementation it guards: per-sweep last-write-wins fails both duplicate-ordering cases, and continue-on-unknown-state fails both reserve cases. The whole-file perturbation sweep still runs clean against the committed tree (eight perturbations, all fail closed), and a one-character edit to the closeout CSV now fails test_csv_and_jsonl_agree[2026-W36-closeout.jsonl].
• Register claims re-read from the source rather than trusted: #3020 115 → 116 checked-in tool snapshots, #3076 116 → 117, mcp-grafana#1080 one added MustTool registration; #3076's 5ea9a0e8… confirmed as refs/pull/3076/head and unreachable from main.
• init/scan reproduced by hand on the deer-flow head at backend/packages/harness (manifest written, scan reaches a decision) and at the repository root (refused_unresolved_scope), and on openai-agents-python at its own recorded pin (same refusal).
• Committed sweep validated against its JSONL: 45 rows each, same order and URLs, no field disagreements, one schema version, no free-text field over 300 characters. Sweep and inventory CSVs re-written with LF terminators to match the committed files; the inventory diff is four rows.

Release-readiness notes

• No user-code import added to default scan paths
• No network access added to default scan paths (the miner is maintainer tooling under benchmark/, already network-touching by design)
• New or changed check IDs are documented in docs/checks.md — none changed
• Report/schema changes are additive or documented in STABILITY.md — none

Closes nothing on its own: #456 stays open for Cut C (calibration, labels, freeze), Cut D (receipts) and Gate 2.

🤖 Generated with Claude Code

[pengfei-threemoonslab]

Expert review completed on 8de35e2. The current 45-row artifacts match exactly, the four newly used upstream pins follow the merge-commit/first-parent convention, the targeted inventory and miner-corpus suites pass, Ruff/compileall/diff-check pass, GitHub CI is green, and the local Agents Shipgate verifier reports complete/passed.

I found two durability gaps in the corpus guards plus three documentation/data consistency issues below. Because this GitHub identity owns the PR, GitHub only permits a COMMENT review; substantively, the P2 items should be addressed before merge.

[pengfei-threemoonslab]

[P2] Bind every accepted results CSV to its JSONL sibling. This loop now treats all CSVs as pin authority, but test_miner_corpus.py validates well-formedness and CSV/JSONL parity only for *-mined.jsonl. The new 2026-W36-closeout pair (and W36-cutb) is intentionally outside that glob, so a half-regeneration can leave this CSV stale while JSONL changes and these stale pins still certify the inventory; corruption in most other rows also bypasses CI. Add a separate all-results integrity enumeration while keeping only the KPI aggregation on *-mined, or validate each sibling JSONL before accepting pins here.

[pengfei-threemoonslab]

Done in 8ff68ef. Split the enumeration in tests/test_miner_corpus.py: _all_run_jsonl_files() (every *.jsonl) now drives well-formedness, CSV/JSONL parity and the LF check, and _mined_jsonl_files() keeps only the KPI aggregation, since the noise bound is meaningful over unselected sweeps alone. All seven committed runs pass unchanged, so the four that were outside the glob — -reeval, -cutb, -closeout, constructed — are load-bearing from this commit.

Two guards keep the split from decaying. test_at_least_one_committed_run_exists now also asserts the mined set is a strict subset, so the two enumerations cannot silently converge; and test_every_committed_results_csv_has_a_jsonl_to_be_checked_against fails on a results CSV with no JSONL sibling, which is the one way a file could still be pin authority while escaping the enumeration entirely. Both perturbed: a one-character edit to the closeout CSV fails test_csv_and_jsonl_agree[2026-W36-closeout.jsonl], and an orphan CSV fails the sibling guard.

[pengfei-threemoonslab]

[P2] Commit the fail-closed perturbations as regression tests. All checked-in inputs are valid, so the suite still passes if this helper regresses to the earlier per-sweep last-write-wins implementation, or if _reserve_claims regresses to skipping an open row that states real_history—the exact two holes fixed on this branch. Refactor the parsers to accept test paths/text and add temporary fixtures for a contradictory duplicate in one sweep (both row orders) and open-plus-origin / misspelled reserve claims. The manual perturbation sweep in the PR description will not protect later edits.

[pengfei-threemoonslab]

Done in 8ff68ef, and you are right that the manual sweep protected nothing — I reverted each implementation to confirm it.

Both parsers now take their input (_swept_pins(sources=...), _reserve_claims(lines=...)), and the row-versus-sweep rule moved out of the test body into _pin_complaints(rows, swept) so a fixture goes through the same code the committed inventory does; the guard is now one assertion that the complaint list is empty. _unsuppliable_origins(claims) does the same for the reserve rule.

Committed perturbations: a contradictory duplicate inside one sweep parametrized over both row orders (only bad-first ever failed, which is exactly why the bug survived), an uncorroborated pin, a resolved subject left unpinned, an open PR reserved with a real origin, and a misspelled reserve origin. Verified by regression rather than by construction — reverting _swept_pins to per-sweep last-write-wins fails both duplicate cases, and reverting _reserve_claims to continue-on-unknown-state fails both reserve cases.

[pengfei-threemoonslab]

[P3] Remove the now-completed instruction from this pinned row. The full base and merge-commit pins are present, but the note still tells the reader to resolve both SHAs and foregrounds the discarded 5ea9a0e8 PR-head abbreviation. State that the close-out resolved the usable head to 8ec62491 instead.

[pengfei-threemoonslab]

Done in 8ff68ef. The note now reads: "Pinned by the close-out sweep at the merge commit 8ec62491 and its first parent bfb59bb7; the walk note also carried 5ea9a0e8, which is the PR branch head and not reachable from the default branch." The instruction is gone and the discarded abbreviation is demoted to the reason the resolution needed stating, which matches the register entry for the same candidate.

[pengfei-threemoonslab]

[P3] This still counts three candidates as three servers. PRs 3020 and 3076 are two changes to the same github-mcp-server; the close-out covers three walked candidates across two MCP-server repositories. This is the same overcount the later findings prose already corrects, so use “three walked candidates” here as well.

[pengfei-threemoonslab]

Done in 8ff68ef — the run-table row now says "the three walked candidates the inventory carried unpinned (two MCP-server repositories)". Same overcount, missed because I fixed the findings prose below and not the row above it.

[pengfei-threemoonslab]

[P3] Reconcile the newly complete table with the surrounding current-state narrative. Lines 327–330 still say the plan “reaches 32” by committing to mine 18 further qualifying candidates, and lines 556–558 still put “Cut B mines the gaps” in the future, even though this table now says 33 sourced / 0 left and the added close-out says sourcing is finished. Update those counts and tense so the reviewer handoff has one current status.

[pengfei-threemoonslab]

Done in 8ff68ef, all three places. The origin-floor paragraph is now past tense and stops restating a count the table above owns ("clearing that floor took 18 further qualifying candidates, more than half of it, which Cut B and the close-out mined. The counts the plan holds now are in the tables above, which are recomputed from the CSV"). The order-of-work line says Cut B and the close-out are both done and names the calibration round as next, and the paragraph under it no longer repeats the status. I also found a third instance you did not flag: the file opened with "so Cut B mines the empty cells", which now reads "could mine" and carries the current status in the same paragraph. One status, at the top, restated nowhere.