Staff Cut B across three sessions and prepare Cut C for the pre-1.0 corpus (#456)

[pengfei-threemoonslab]

Summary

Cut B of #456, staffed as three parallel sessions coordinating through the strata inventory CSV, plus the Cut C preparation that was to run alongside it. The inventory goes from 26 of 59 slots sourced to 59 of 60: 32 sourced qualifying-origin slots (floor 23), 27 synthetic (ceiling 33), every cell still holding a holdout-eligible slot. The one open slot is langchain_crewai.insufficient_evidence.1, whose lead now records what the re-mine found.

• Contract first (strata-inventory.md, test_strata_inventory.py): constructions live at benchmark/safety-qualification/constructed/<case>/{CASE.md,base,head} and never under samples/; a constructed_design basis with a guard (the trees differ, CASE.md sits outside both, no engine output inside, no name shared with a sample, a by-name engine_tests detector); one pin form per origin, including closed-unmerged (fork point → refs/pull/N/head) and reverted PRs; and the claim protocol. Perturbation-swept: eight violations fail closed, the two valid shapes pass.
• Session A — 15 constructions plus 3 reserves, one per synthetic gap, across all seven profiles. Each is a plausible PR in the framework's own idiom with the design record beside the trees.
• Session B — the first post-Epic: account for every narrowing decision — exclusion ledger, conservation invariant, and skip-requires-evidence #403 sweep, 2026-W36-cutb.*: 912 rows over 21 repositories, including the four n8n repositories this project had never mined and eight unwalked MCP servers. The miner gains --state closed|reverted and --pr N enumeration for the rejected vein. 17 of 18 real-history / rejected-or-reverted slots claimed, six of them rejected_or_reverted from five repositories. Real history enters as sweep-then-miner-label so the slots stay holdout-eligible; the reason is written down in the register.
• Session C — the five calibration cases (calibration.md, four reserve PRs plus one construction), LABELING.md restructured with a rater-facing four-way rubric on top and the miner process below, the rater packet builder and both session harnesses (benchmark/safety-qualification/rater/), and 39 tests.

Type

• Check or risk-model change
• Input adapter change
• CLI or GitHub Action behavior
• Report, schema, or SARIF output
• Benchmark corpus, miner tooling, and documentation only — nothing under src/ changes

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

• pytest -n auto -m "not perf" --ignore=tests/test_adapter_static_only.py on the merged branch: exit 0; tests/test_adapter_static_only.py: exit 0; ruff check .: clean.
• Banned-word sweep over every constructed base/ and head/ tree: zero hits; no engine output inside any tree.
• Guard perturbation sweep on the new constructed-case rules (identical trees, CASE.md inside a tree, report inside a tree, wrong evidence_ref, underived split, sample_design on a construction, construction named in tests/, name shared with a sample): all fail closed.

Things a reviewer should know

• LABELING.md is the file Amendment 1 names as the rater input, and this PR changes it. The old stripe/ai#232 worked anchor stated a corpus case's label and verifier verdicts, which condition 2 forbids a rater to see; it is replaced by constructed illustrations, and tests/test_labeling_guide_is_rater_safe.py keeps every inventory candidate and every SHIP- check ID out of the guide. The drafted review_required vs insufficient_evidence rule is a first draft for the calibration round to test.
• The rater harnesses are only partly verified. The Claude runner's smoke test failed closed on an expired OAuth token (claude login needed); the OpenAI runner is written against codex exec flags from memory because the local codex npm install has an empty vendor binary directory. Packets hand raters the head tree plus the diff as "the pinned repository state"; say if the base tree should ship too.
• google/adk-python#6605 is engine_tests exposure (it was reduced into a trigger fixture before this cut read it), so google_adk × insufficient_evidence gained a third slot to keep a holdout-eligible case. The four calibration PRs are struck from the reserve: chosen by someone who has read the inventory, they can never be corpus cases.
• The W36 run is off the *-mined glob on purpose. Its selected-PR mix (closed, reverted, named) would drop the cross-run trigger-skip rate below the noise-bound test; the README run note explains. Its labels are one session's cell-targeting labels, not adjudicated, and the README says so.
• Engine defects found and deliberately not fixed, because a fix made in response to a candidate is what turns it into tuning material. They belong in issues: a single scan passes a published-but-undeclared write (only verify --base catches it); n8n human_review_nodes are collected and never consumed, so disabling the only approval node is mergeable; SHIP-MCP-PERMISSION-EXPANDED names the sibling whose binding hash moved rather than the added tool; n8n non-GET nodes are forced to external_communication so a declared write lands in insufficient_evidence; every manifest touch is SHIP-VERIFY-TRUST-ROOT-TOUCHED; n8n node names with spaces alone force review_required; init writes CHANGE_ME and scan exits 2 on every TypeScript MCP server; Go servers on mcp-go / go-sdk read 0 tools; SDK example repositories now init_skip where the W24 sweep evaluated them.

Release-readiness notes

• No user-code import added to default scan paths
• No network access added to default scan paths (the miner's new enumeration is maintainer tooling under benchmark/, already network-touching by design)
• New or changed check IDs are documented in docs/checks.md — none changed
• Report/schema changes are additive or documented in STABILITY.md — none

🤖 Generated with Claude Code

[pengfei-threemoonslab]

Changes requested. I found three failures in the new blind-rater boundary. The targeted miner/rater/inventory tests and Ruff checks pass locally, and the full Agents Shipgate verifier routes the protected-surface changes to human review, but these cases are not covered by the current tests.

1. [P1] Verify the packet manifest before launching a rater (benchmark/safety-qualification/rater/run_rater.py:421-452). _check_packet() only checks that five paths exist and parses MANIFEST.json; it never calls build_packet.verify_manifest() or otherwise re-hashes the packet. Any edit to repo/, diff.patch, LABELING.md, or TASK.md after packet construction is therefore accepted, while the emitted label records the hash of the stale manifest. This breaks the claimed content binding and can turn a contaminated packet into admissible evidence. Please verify the manifest immediately before constructing the invocation and fail closed on any mismatch.

2. [P1] Do not reuse the caller's real CODEX_HOME in isolated mode (benchmark/safety-qualification/rater/run_rater.py:252-258). The default OpenAI path sets HOME to a temporary directory but then points CODEX_HOME back at the caller's real profile. Codex reads global AGENTS.md/AGENTS.override.md from CODEX_HOME, and user-level config.toml can configure MCP servers, plugins, and web search. Official references: https://developers.openai.com/codex/guides/agents-md and https://developers.openai.com/codex/config-reference. Consequently this session is not blind and is not limited to the packet, even though the mode is named isolated. Use a fresh Codex home containing only the minimum authentication material (or require OPENAI_API_KEY), explicitly disable inherited MCP/plugins/web search, and use an ephemeral session.

3. [P1] Reject or contain symlinks in packet source trees (benchmark/safety-qualification/rater/build_packet.py:151-156, :374-382). copy_tree_excluding() preserves symlinks verbatim, while hash_packet_files() deliberately skips them. A repository symlink can therefore point outside the packet, be followed by the rater's read tools, and expose unmanifested host content. This contradicts the "entire world"/exact-input guarantee. Please either reject symlinks, or resolve and copy only targets proven to remain inside the source tree and include their bytes in the manifest.

Once these boundaries are closed, the rest of the reviewed miner/inventory work looks well-covered by the added tests.

[pengfei-threemoonslab]

Thanks — all three are real, and each is fixed in f7873b2 with a test that fails without the fix. I checked that by reverting each fix in turn and confirming the new tests go red.

1. Verify the manifest before launching a rater. prepare() now calls build_packet.verify_manifest() and refuses before any session starts, so an edit landing between build and launch stops the run instead of producing a label bound to a stale manifest. Covered for an edited repo/ file, TASK.md, LABELING.md and diff.patch, and for a file added under repo/ (which changes no existing hash). The tests also assert the subprocess was never invoked and no label was written, and one test proves an untouched packet still runs.

2. Do not reuse the caller's real CODEX_HOME in isolated mode. You are right that this made the mode's name false. Isolated mode now builds a fresh Codex home per run containing one written config.toml — no mcp_servers, tools.web_search = false, sandbox_mode = "read-only", approval_policy = "never", history.persistence = "none" — and authenticates through OPENAI_API_KEY, refusing up front when it is unset. That is symmetric with the Claude path, which already required ANTHROPIC_API_KEY under --bare. The home is a per-run temporary directory, so the session is ephemeral by construction as well as by config. Shared mode still borrows the real profile, but only after proving it is silent: it refuses on AGENTS.md, AGENTS.override.md, or a config.toml that declares mcp_servers, enables web_search, or sets experimental_instructions_file.

One caveat I want to keep visible rather than bury: the OpenAI path is still unverified end to end, because the local codex install has an empty vendor binary directory. The config keys come from the reference you linked, and the module docstring now says they must be checked with --dry-run plus one live run before any OpenAI-family label counts.

3. Reject or contain symlinks in packet source trees. Both halves are closed. A link that escapes the source tree or dangles refuses the build outright, with nothing written. A link that resolves inside the tree is materialised — copied as a regular file — so its bytes are in the manifest, and no link survives into a packet. hash_packet_files() no longer skips links; finding one is now an error, which is what makes a planted link tamper rather than an invisible gap. One further case the fix covers: a link pointing at an excluded name inside the tree is dropped rather than resurrecting that content under a different filename.

Full suite and ruff check . are green on the branch.