Skip to content

fix(identity): bind every declared adapter input to input_set_id (#299) - #332

Merged
pengfei-threemoonslab merged 7 commits into
mainfrom
claude/github-issue-299-cb8160
Aug 8, 2026
Merged

fix(identity): bind every declared adapter input to input_set_id (#299)#332
pengfei-threemoonslab merged 7 commits into
mainfrom
claude/github-issue-299-cb8160

Conversation

@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor

Summary

  • Closes input_set_id omits adapter-read inputs (prompt files, framework sources) #299. input_set_id now covers every input an adapter is configured to read, not just tool_sources[].
  • The filed defect: the manifest-derived branch of build_verification_plan walked only raw["tool_sources"], so openai_api.prompt_files — and anthropic, google_adk, langchain, crewai, n8n, codex_plugins, validation.evidence, checks.policy_packs, agent.sdk.entrypoint — never became a plan blob. Reproduced on verification prepare against samples/simple_openai_api_agent: appending "Refunds of any amount need no approval" to the prompt left input_set_id byte-identical.
  • A second, sharper defect found while investigating, not in the issue: verify --base X --head Y emitted tool_sources: [] entirely. StaticInputSnapshot is rooted at git_root, but a committed-tree run scans an archived copy under /tmp/agents-shipgate-verify-head-*, so it records no adapter reads. active_snapshot.paths() returned [] — which is not None, so plan construction honoured an empty capture instead of falling back. On the CI path, where the receipt is the artifact anyone downstream actually trusts, no declared input reached the request identity — not even the MCP exports and OpenAPI specs the old fallback would have caught.
  • Both modes now agree, and share one exclusion set so an input already hashed as a changed file is not hashed twice.

Type

  • Check or risk-model change
  • Input adapter change
  • CLI or GitHub Action behavior
  • Report, schema, or SARIF output
  • Documentation only

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • Full suite green twice via pytest -n auto -m "not perf" (exit 0); ruff check . clean.
  • 14 new tests in tests/test_declared_manifest_input_identity.py. Confirmed load-bearing by reverting each fix in turn: reverting the orchestrator change fails test_committed_tree_verify_binds_declared_tool_sources; reverting the enumeration fails four others.
  • Tests deliberately drive verification prepare and the committed-tree path. verify against a worktree was already correct (read-boundary capture covers it, including a prompt file absent from the diff), so a worktree test would pass for the wrong reason — the issue's triage note called this out.
  • Dogfooded: verify --base main --head HEAD on this repo now binds .agents/plugins/marketplace.json where it previously bound nothing; receipt input_set_id matches the plan; decision passed.
  • Error paths exercised by hand: unparseable manifest on a committed-tree run still routes config_error/exit 2 and still writes a plan carrying the config blob; an out-of-root declared path routes input_parse_error/exit 3.

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md — n/a, no check IDs touched
  • Report/schema changes are additive or documented in STABILITY.md — no schema change; plan.inputs.tool_sources gains entries, not fields

Notes for the reviewer

Why the path table is derived, not hand-kept. schemas/manifest/declared_paths.py walks raw yaml.safe_load output (identity must stay constructible for a manifest that fails validation) and recognizes paths by two rules: a path: key anywhere in a path-bearing block, plus any key named after a field whose type carries ArtifactPathConfig. That second rule is load-bearing — _parse_artifact_entries also accepts a bare string (tools: [tools/openai.json]), which a path:-only walk misses. The key set is read off the pydantic models at import time, so a new artifact list, or a whole new framework block, is covered with no edit here. Only non-ArtifactPathConfig string paths need naming by hand (_UNTYPED_PATH_FIELDS): agent.sdk.entrypoint, both prompt_files, and tool_sources (registered for the path: rule alone).

Deliberately not inputs, to stay symmetric with what read-boundary capture sees: output.directory (an output), organization.audit.registry (existence-tested, never read), baseline.audit_log (resolved against the baseline file, not an adapter input).

Two things worth a second opinion:

  1. One new way to fail, on purpose. A declared path outside the input root cannot be hashed portably, so it is now rejected rather than dropped. Realistically only reachable via agent.sdk.entrypoint with no openai_agents_sdk tool source, since resolve_input_path rejects the rest at scan time. Fail-closed felt right — a silently unbound declared input is the class this issue is about — but it will hard-fail a manifest that verifies today. My first attempt raised ValueError, which surfaced as internal_error/exit 4 ("file an issue"); that is the wrong routing for something the user must fix in their manifest, hence InputParseError.
  2. input_set_id and request_id values move for any manifest declaring framework inputs, and for every --head run. Receipts minted before and after this change cannot be compared by ID. Sample expected/report.json goldens are unaffected (they carry input_set_id: nullscan builds no plan).

Generalizable bug class, worth remembering beyond this PR: when a code path selects between "observed" and "declared" on x is None, an observer that is inert rather than absent silently wins with an empty result. Check that the observer could have seen anything at all, not just that it exists.

🤖 Generated with Claude Code

@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-299-cb8160 branch from 64751bb to 545e291 Compare August 7, 2026 05:54

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two actionable findings from the review; details are inline.

# branches must reach every adapter input: a path that lands in neither can
# change bytes while ``input_set_id`` stays byte-identical.
candidate_input_paths = (
_manifest_declared_input_paths(config_path=config_path, input_root=input_root)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Capture transitive adapter reads. This fallback enumerates only paths declared directly in shipgate.yaml. Inputs discovered while parsing an entrypoint—such as a Google ADK McpToolset inventory or OpenAPI spec—remain absent from plan.inputs.tool_sources. I reproduced two clean heads whose inventories/mcp-tools.json bytes differed (only by a trailing newline) but whose prepared input_set_id was identical; the plan bound agent.py, the eval set, and the function inventory, but not the MCP inventory. Please capture reads against the evaluated input root or statically expand transitive dependencies before building the plan, with a regression test for this case.

no_heuristics=no_heuristics,
)
except InputParseError as exc:
typer.echo(f"Input parsing error: {exc}", err=True)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the agent-mode error envelope. This new catch emits only prose and exits. With AGENTS_SHIPGATE_AGENT_MODE=1, an out-of-root declared input produces no structured input_parse_error, next_action, or next_actions, contrary to the CLI agent contract. Please emit the same agent-mode error payload used by verify before raising typer.Exit(3).

pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
…rations

Review follow-up on #332.

Enumerating the manifest reaches only what the manifest names. An input
discovered while parsing something it names — a Google ADK `McpToolset`
inventory, an OpenAPI spec constructed inside `agent.py`, a sub-agent
config — is invisible to that walk, so two trees whose adapters read
different bytes still shared an `input_set_id`. Reproduced on the ADK
sample: the plan bound `agent.py`, the eval set, and the function
inventory, but not `inventories/mcp-tools.json`, and a trailing newline
on it left identity byte-identical.

Both remaining producers now observe the read boundary instead. A
committed-tree `verify` is snapshotted against the archived tree it
scans, which the worktree-bound snapshot could never see, and
`verification prepare` loads sources — statically, deciding nothing — to
record what they open. Committed-tree and worktree runs of the same tree
now bind the same set, asserted as an invariant rather than per-file.
The declared-path enumeration survives only as the fallback for a plan
built with no snapshot at all.

A committed-tree run therefore has two snapshots alive, and each external
input must belong to exactly one of them. The first attempt let the
archived scan read the baseline and policy packs through no snapshot at
all, silently defeating the tamper check that
`test_archived_verify_rejects_external_baseline_change_after_scan`
exists to enforce; the second let both watch the same directory, and the
second re-validation then failed on a change the first legitimately
allowed. The worktree snapshot now binds them before the archived scan
starts, which also widens the guarded window: it begins before the scan
rather than at the scan's first read.

`verification prepare` reads inputs now, so it fails on a manifest whose
inputs cannot be loaded — the same condition under which `verify` fails,
and exactly when a prepared plan could not honestly claim an input set.
Its errors carry the agent-mode envelope (`input_parse_error` /
`config_error` with `next_action` and `next_actions`, per
docs/errors.json) instead of prose alone.

With capture in place, a declared path that no adapter opens is correctly
not an input, so the out-of-root rejection now applies only to the
declared fallback; its test moved to target that path directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Both findings were real and reproduced before fixing. Addressed in 84db5989.

[P1] Capture transitive adapter reads

Reproduced exactly as described on samples/google_adk_agent: agent.py constructs McpToolset(inventory_path="inventories/mcp-tools.json"), nothing in shipgate.yaml names that file, and a trailing newline on it left the prepared input_set_id byte-identical. The plan bound agent.py, the eval set and the function inventory — and, as you note, not the MCP inventory.

I took the first of your two routes. Statically expanding transitive references would mean re-deriving each adapter's discovery logic in the identity builder and keeping it in step forever; the read boundary already knows the answer exactly, and it is the same boundary the receipt is bound to.

Both remaining producers now observe it:

  • committed-tree verify is snapshotted against the archived tree it scans (orchestrator.py). The outer snapshot is bound to the worktree and could never see a read under /tmp/agents-shipgate-verify-head-*, which is why this mode saw nothing.
  • verification prepare loads sources under a snapshot and records what they open. Loading is static, local, network-free and decision-free, and the scan phases after it make no further declared-input reads, so "prepare does not evaluate policy" still holds.

Committed-tree and worktree runs of the same tree now bind the same set, asserted as an invariant (test_committed_tree_verify_agrees_with_the_worktree_on_the_input_set) rather than as a per-file list, plus test_prepare_binds_a_transitively_referenced_input for your case directly. Declared-path enumeration survives only as the fallback for a plan built with no snapshot at all.

Same bug class showed up in a second adapter: on this repo prepare went from binding 1 file to 3, picking up the Codex marketplace entry's transitive plugin.json and SKILL.md.

One implementation note in case it saves someone time later: the snapshot root must be .resolve()d. On macOS the temp tree is reached through /var while every adapter resolves its base dir, so an unresolved root makes contains() false for every path beneath it and capture silently returns empty — the same inert-observer failure as the original defect.

[P2] Preserve the agent-mode error envelope

Fixed via emit_agent_mode_error_action, so input_parse_error carries both next_action and next_actions per docs/errors.json. I also added the config_error route, which the capture step made reachable, and a test asserting the envelope shape rather than the prose.

A guard I nearly broke, and what it cost

My first cut at the archived-head snapshot defeated a TOCTOU check, and the existing suite caught it: test_archived_verify_rejects_external_baseline_change_after_scan. Swapping the active snapshot meant externally supplied inputs — baseline, policy packs, comparison report, which keep their worktree location even for a committed-tree run — were read through no snapshot, so nothing held their pre-scan bytes.

The obvious repair, giving the head snapshot those external paths too, traded one failure for another: two snapshots re-validating the same directory, where the second trips on a change the first legitimately allowed (test_finalized_snapshot_keeps_deleted_baseline_in_plan).

The working rule is ownership — each external input belongs to exactly one snapshot. The worktree snapshot binds them before the archived scan begins, which also widens the guarded window rather than narrowing it: it now starts before the scan instead of at the scan's first read. That constraint is recorded in the CHANGELOG, since nothing in the code makes it obvious that adding a second watcher would break it.

Consequences worth a second look

  • prepare now fails on a manifest whose inputs cannot be loaded, where it previously emitted a plan. I believe that is correct — it is exactly when a prepared plan cannot honestly claim an input set, and verify fails on the same tree — but it is a behavior change to a shipped command, so it is called out in the CHANGELOG and docs/verification-reproducibility.md rather than left implicit. Say the word if you would rather it degrade to the declared enumeration instead.
  • The out-of-root rejection narrowed. With capture, a declared path no adapter opens is correctly not an input, so prepare no longer rejects it; the guard now applies only to the declared fallback. Its test moved to target that path directly instead of asserting behavior that no longer exists.

Full suite green (pytest -n auto -m "not perf", 0 failures), ruff and compileall clean, branch rebased on current main.

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One newly reproduced blocking finding from the follow-up review; details are inline.

# snapshot is not left active past the scan; it is the record of
# which paths were read, not the reader for the receipt.
if head_snapshot_token is not None:
reset_static_input_snapshot(head_snapshot_token)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Build the receipt from the captured bytes. Resetting the head snapshot here means _write_artifacts later supplies only head_snapshot.paths() while build_blob reopens the mutable archived files. I reproduced a mutation immediately after snapshot finalization: the resulting plan hashed the post-scan agent.py bytes referencing inventories/mcp-tools-2.json, but its tool_sources still bound the pre-scan inventories/mcp-tools.json and omitted the new inventory. The receipt can therefore attest to bytes the report never evaluated. Please keep the finalized snapshot active for plan/blob construction or pass its cached bytes/blobs forward; the same capture-reset-reread window exists in verification prepare, where _captured_input_paths finalizes and deactivates its snapshot before build_verification_plan reopens the files. Add mutation-after-capture regressions for both paths.

pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
Review follow-up on #332.

Capture recorded which paths the adapters opened and then released the
snapshot, leaving `build_verification_plan` to reopen those files to hash
them. The path list and the blob hashes therefore came from two different
instants. A file rewritten in between is attested at its new content while
`tool_sources` still lists what the old content pointed at — so the plan
can bind a pre-scan `inventories/mcp-tools.json`, omit the inventory the
post-scan `agent.py` actually names, and hash `agent.py` at bytes the
report never evaluated. Reproduced on `prepare`: captured `agent.py` was
sha256:7465..., the emitted plan recorded sha256:040c....

My earlier note here claimed the reset was safe because the archived tree
is a private temporary directory. That reasoned about who can write the
files and missed the actual defect, which is that the two halves of the
plan are taken at different times regardless of the threat model.

Plan construction now runs under the finalized snapshot on both paths:
`verify` re-activates the head snapshot around the plan build, and
`prepare`'s capture became a context manager that holds the snapshot open
for it. Blobs are hashed from captured bytes and never reopened.

That exposed a second-order requirement. `_blobs` skips a path the
snapshot contains but never read, so binding the snapshot without also
binding the changed files silently dropped every changed file no adapter
opens — verified: `changed_files` goes to empty. The worktree path already
preloaded changed files for exactly this reason; that block is now
`_bind_changed_files` and is applied to the archived tree and to
`prepare` as well.

Both regressions fail against the unfixed code, checked by reverting each
activation independently, and the committed-tree one also asserts an
unrelated changed file survives capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Confirmed and fixed in fc68e48f. You're right, and the reasoning I left in that comment was the problem.

I had noticed the reset window and talked myself out of it with "plan construction hashes the archived files directly … the archive is a private temp dir that nothing else writes." That argues about who can write the files, which is not the defect. The defect is that the path list and the blob hashes are taken at two different instants, so the plan is internally inconsistent no matter who can write — exactly as you show, tool_sources describing what the pre-scan agent.py pointed at while agent.py's own blob attests to post-scan bytes.

Reproduced on prepare before fixing: captured agent.py was sha256:7465…, the emitted plan recorded sha256:040c….

Fix

Plan construction now runs under the finalized snapshot on both paths, so blobs are hashed from captured bytes and the files are never reopened:

  • verify re-activates the head snapshot around build_verification_plan and releases it again afterwards, so the worktree snapshot stays the reader for the externally supplied inputs it owns.
  • prepare's capture became a context manager that holds the snapshot open across the plan build, instead of returning a path list after deactivating.

Second-order requirement this exposed

_blobs skips a path the snapshot contains but never read. So binding the snapshot without also binding the changed files silently dropped every changed file no adapter opens — a README, an unrelated module. Verified rather than assumed: with the archived bind removed, plan.inputs.changed_files goes to empty.

The worktree path already preloaded changed files for exactly this reason; that block is now _bind_changed_files and is applied to the archived tree and to prepare as well. The committed-tree regression asserts an unrelated note.txt survives capture, so this cannot silently reappear.

Regressions

test_prepare_hashes_the_captured_bytes_not_a_later_rewrite and test_committed_tree_verify_hashes_the_captured_bytes_not_a_later_rewrite, both mutating after snapshot finalization via a StaticInputSnapshot.finish hook. Each was checked against the unfixed code by reverting its activation independently — both fail there and pass with the fix. The committed-tree one deliberately targets the archived copy rather than the worktree one, since the worktree snapshot finalizes first.

Full suite green (pytest -n auto -m "not perf", 0 failures), ruff and compileall clean, branch current with main.

Running tally of behaviour changes, for the final read-through

  1. verification prepare reads inputs, so it fails on a manifest whose inputs cannot load (previously emitted a plan). Same condition under which verify fails.
  2. prepare routes its errors — exit 2/3 with the agent-mode next_action/next_actions envelope — instead of printing a traceback.
  3. The out-of-root declared-path rejection applies only to the no-snapshot fallback; with capture, a declared path no adapter opens is correctly not an input.
  4. input_set_id / request_id values move. No schema change.

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One newly reproduced blocking identity finding from the latest follow-up review; details are inline.

candidate = resolved_root / relative
if candidate.is_file() and not candidate.is_symlink():
snapshot.read_bytes(candidate, max_bytes=_MAX_CHANGED_FILE_BYTES)
snapshot.finish()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Bind the explicit baseline and comparison inputs before finalizing. Keeping this snapshot active through plan construction makes _optional_blob treat any contained-but-unread path as absent, but _captured_inputs currently reads adapter/policy inputs and changed files only. I reproduced two clean preparations on the same HEAD, one with --baseline baseline-a.json --diff-from comparison-a.json and one with different B files: both exited successfully, emitted inputs.baseline = null and inputs.diff_from = null, and produced the same input_set_id. The prepared request therefore ignores two explicit verification inputs. Please pass these paths into the capture context and bind their bytes before finish() (including correct external ownership for the archived-head case), with regressions asserting the blobs are present and that changing either input changes the identity.

pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
…rations

Review follow-up on #332.

Enumerating the manifest reaches only what the manifest names. An input
discovered while parsing something it names — a Google ADK `McpToolset`
inventory, an OpenAPI spec constructed inside `agent.py`, a sub-agent
config — is invisible to that walk, so two trees whose adapters read
different bytes still shared an `input_set_id`. Reproduced on the ADK
sample: the plan bound `agent.py`, the eval set, and the function
inventory, but not `inventories/mcp-tools.json`, and a trailing newline
on it left identity byte-identical.

Both remaining producers now observe the read boundary instead. A
committed-tree `verify` is snapshotted against the archived tree it
scans, which the worktree-bound snapshot could never see, and
`verification prepare` loads sources — statically, deciding nothing — to
record what they open. Committed-tree and worktree runs of the same tree
now bind the same set, asserted as an invariant rather than per-file.
The declared-path enumeration survives only as the fallback for a plan
built with no snapshot at all.

A committed-tree run therefore has two snapshots alive, and each external
input must belong to exactly one of them. The first attempt let the
archived scan read the baseline and policy packs through no snapshot at
all, silently defeating the tamper check that
`test_archived_verify_rejects_external_baseline_change_after_scan`
exists to enforce; the second let both watch the same directory, and the
second re-validation then failed on a change the first legitimately
allowed. The worktree snapshot now binds them before the archived scan
starts, which also widens the guarded window: it begins before the scan
rather than at the scan's first read.

`verification prepare` reads inputs now, so it fails on a manifest whose
inputs cannot be loaded — the same condition under which `verify` fails,
and exactly when a prepared plan could not honestly claim an input set.
Its errors carry the agent-mode envelope (`input_parse_error` /
`config_error` with `next_action` and `next_actions`, per
docs/errors.json) instead of prose alone.

With capture in place, a declared path that no adapter opens is correctly
not an input, so the out-of-root rejection now applies only to the
declared fallback; its test moved to target that path directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
Review follow-up on #332.

Capture recorded which paths the adapters opened and then released the
snapshot, leaving `build_verification_plan` to reopen those files to hash
them. The path list and the blob hashes therefore came from two different
instants. A file rewritten in between is attested at its new content while
`tool_sources` still lists what the old content pointed at — so the plan
can bind a pre-scan `inventories/mcp-tools.json`, omit the inventory the
post-scan `agent.py` actually names, and hash `agent.py` at bytes the
report never evaluated. Reproduced on `prepare`: captured `agent.py` was
sha256:7465..., the emitted plan recorded sha256:040c....

My earlier note here claimed the reset was safe because the archived tree
is a private temporary directory. That reasoned about who can write the
files and missed the actual defect, which is that the two halves of the
plan are taken at different times regardless of the threat model.

Plan construction now runs under the finalized snapshot on both paths:
`verify` re-activates the head snapshot around the plan build, and
`prepare`'s capture became a context manager that holds the snapshot open
for it. Blobs are hashed from captured bytes and never reopened.

That exposed a second-order requirement. `_blobs` skips a path the
snapshot contains but never read, so binding the snapshot without also
binding the changed files silently dropped every changed file no adapter
opens — verified: `changed_files` goes to empty. The worktree path already
preloaded changed files for exactly this reason; that block is now
`_bind_changed_files` and is applied to the archived tree and to
`prepare` as well.

Both regressions fail against the unfixed code, checked by reverting each
activation independently, and the committed-tree one also asserts an
unrelated changed file survives capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-299-cb8160 branch from fc68e48 to 89f5bb4 Compare August 7, 2026 21:30
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Reproduced and fixed in 89f5bb47. This one is a regression I introduced with the previous fix, not a pre-existing gap.

Activating the snapshot for plan construction turned binding from an optimization into an obligation: under an active snapshot both _blobs and _optional_blob read a path that is contained but never read as absent. I named exactly that trap for changed_files, fixed that one case, and did not sweep the other consumers — so --baseline and --diff-from disappeared. Your repro reproduces verbatim: baseline: null, diff_from: null, identical input_set_id across different A/B files.

Audited rather than patched at the reported site

_blobs and _optional_blob are the only two helpers that drop unread paths. Every call to them inside build_verification_plan:

plan input helper drops unread? bound
config build_blob no — has() with direct-read fallback
tool_sources _blobs yes adapter reads
policy_packs _blobs yes ✓ now
changed_files _blobs yes
baseline, diff_from _optional_blob yes ✓ now
worktree overlay has() n/a
engine digest sha256_file no — same fallback

That is the complete inputs_payload. build_blob and sha256_file gate on has() and fall back to a direct read, so they cannot drop — worth stating explicitly since it is why the config blob survived this bug.

Two details in the fix

  • External ownership, as you called out: the comparison report is never mapped into the archived tree, so on a committed-tree preparation it stays outside the evaluated root and must be declared an external input — otherwise the snapshot refuses to read it at all and the blob vanishes for a second, different reason.
  • The archive directory is now resolved once it exists. macOS reaches it through /var while every derived path resolves to /private/var, and the snapshot matches lexically, so the two spellings make contains() false for inputs plainly inside the tree. Third time this alias has bitten this change; resolving at the source kills the class.

Regressions

test_prepare_binds_the_explicit_baseline_and_comparison_report, parametrized over worktree and committed-tree. Asserts both blobs are present with the right logical paths and that swapping either input moves input_set_id. Both variants fail with the binding removed — verified, not assumed.

One trap worth recording for anyone writing similar tests: the output must land outside the repository. My first attempt wrote plans into the workspace, and the first run's artifacts became untracked changed files that moved identity on their own — which would have made the test pass for entirely the wrong reason.

Rebase

main moved to #331 (verifier artifact lifecycle) mid-review. Rebased onto 23c49317; only CHANGELOG.md conflicted. #331 also edits cli/verify/orchestrator.py, so I checked that merge semantically rather than trusting it: the two are disjoint — #331 swaps _clear_trusted_handoff for the shared clear_verifier_route_artifacts at the run entry points, mine is in the snapshot/plan path, and its out_dir cleanup cannot interact with the head snapshot because that snapshot excludes out_dir. Both changes verified present after the merge.

Full suite green on the rebased tree (pytest -n auto -m "not perf", 0 failures), ruff and compileall clean.

Behaviour changes accumulated across this PR

  1. verification prepare reads inputs, so it fails on a manifest whose inputs cannot load (previously emitted a plan). Same condition under which verify fails.
  2. prepare routes its errors — exit 2/3 with the agent-mode next_action/next_actions envelope — instead of printing a traceback.
  3. The out-of-root declared-path rejection applies only to the no-snapshot fallback.
  4. input_set_id / request_id values move. No schema change.

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A new blocking identity-integrity finding from the latest review is attached inline.

token = activate_static_input_snapshot(snapshot)
try:
try:
resolved = _prepare_scan(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Parse the manifest from the captured bytes. The snapshot is active here, but _prepare_scan() receives no manifest_text; load_manifest_with_positions() therefore first parses the manifest through direct Path.read_text() and only then rereads it through read_static_input_text() for positions. A mutation between those reads makes _load_inputs() follow the old manifest while build_verification_plan() hashes the newer cached manifest. I reproduced this in both verification prepare and committed-tree verify: both exited successfully, and the latter produced a valid terminal receipt whose config SHA covered a manifest naming agent-two.py while tool_sources still bound agent.py from the earlier manifest. Please read the manifest once through the active snapshot and pass those exact bytes/text into _prepare_scan() and the archived run_scan() path, with a regression that mutates the manifest after the first parse.

pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
…rations

Review follow-up on #332.

Enumerating the manifest reaches only what the manifest names. An input
discovered while parsing something it names — a Google ADK `McpToolset`
inventory, an OpenAPI spec constructed inside `agent.py`, a sub-agent
config — is invisible to that walk, so two trees whose adapters read
different bytes still shared an `input_set_id`. Reproduced on the ADK
sample: the plan bound `agent.py`, the eval set, and the function
inventory, but not `inventories/mcp-tools.json`, and a trailing newline
on it left identity byte-identical.

Both remaining producers now observe the read boundary instead. A
committed-tree `verify` is snapshotted against the archived tree it
scans, which the worktree-bound snapshot could never see, and
`verification prepare` loads sources — statically, deciding nothing — to
record what they open. Committed-tree and worktree runs of the same tree
now bind the same set, asserted as an invariant rather than per-file.
The declared-path enumeration survives only as the fallback for a plan
built with no snapshot at all.

A committed-tree run therefore has two snapshots alive, and each external
input must belong to exactly one of them. The first attempt let the
archived scan read the baseline and policy packs through no snapshot at
all, silently defeating the tamper check that
`test_archived_verify_rejects_external_baseline_change_after_scan`
exists to enforce; the second let both watch the same directory, and the
second re-validation then failed on a change the first legitimately
allowed. The worktree snapshot now binds them before the archived scan
starts, which also widens the guarded window: it begins before the scan
rather than at the scan's first read.

`verification prepare` reads inputs now, so it fails on a manifest whose
inputs cannot be loaded — the same condition under which `verify` fails,
and exactly when a prepared plan could not honestly claim an input set.
Its errors carry the agent-mode envelope (`input_parse_error` /
`config_error` with `next_action` and `next_actions`, per
docs/errors.json) instead of prose alone.

With capture in place, a declared path that no adapter opens is correctly
not an input, so the out-of-root rejection now applies only to the
declared fallback; its test moved to target that path directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pengfei-threemoonslab added a commit that referenced this pull request Aug 7, 2026
Review follow-up on #332.

Capture recorded which paths the adapters opened and then released the
snapshot, leaving `build_verification_plan` to reopen those files to hash
them. The path list and the blob hashes therefore came from two different
instants. A file rewritten in between is attested at its new content while
`tool_sources` still lists what the old content pointed at — so the plan
can bind a pre-scan `inventories/mcp-tools.json`, omit the inventory the
post-scan `agent.py` actually names, and hash `agent.py` at bytes the
report never evaluated. Reproduced on `prepare`: captured `agent.py` was
sha256:7465..., the emitted plan recorded sha256:040c....

My earlier note here claimed the reset was safe because the archived tree
is a private temporary directory. That reasoned about who can write the
files and missed the actual defect, which is that the two halves of the
plan are taken at different times regardless of the threat model.

Plan construction now runs under the finalized snapshot on both paths:
`verify` re-activates the head snapshot around the plan build, and
`prepare`'s capture became a context manager that holds the snapshot open
for it. Blobs are hashed from captured bytes and never reopened.

That exposed a second-order requirement. `_blobs` skips a path the
snapshot contains but never read, so binding the snapshot without also
binding the changed files silently dropped every changed file no adapter
opens — verified: `changed_files` goes to empty. The worktree path already
preloaded changed files for exactly this reason; that block is now
`_bind_changed_files` and is applied to the archived tree and to
`prepare` as well.

Both regressions fail against the unfixed code, checked by reverting each
activation independently, and the committed-tree one also asserts an
unrelated changed file survives capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-299-cb8160 branch from 89f5bb4 to 3c8060d Compare August 7, 2026 23:40
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Confirmed and fixed in 3c8060d8.

load_manifest_with_positions parses the manifest twice — a direct Path.read_text for the model, then the snapshot for positions. The worktree path has always dodged this by passing its captured worktree_manifest_text; the committed-tree path passed None, and the prepare capture I added earlier in this branch passed nothing either. Both now read the manifest once through the active snapshot and hand those exact bytes to _prepare_scan / run_scan.

I also resolve the archived tree at creation rather than at each use, so the snapshot's lexical matching sees one spelling. That is the same /var vs /private/var alias that has now bitten this change four times; resolving at the source ends the class rather than patching each site.

Two correct outcomes, so the test asserts the invariant

With the manifest read through the snapshot, prepare fails closedfinish() catches the rewrite and exits 3 — while committed-tree verify emits a self-consistent plan. Both are right, so test_the_scan_and_the_plan_agree_on_one_manifest asserts what must never happen (succeeding with a plan that disagrees with the manifest the scan followed) rather than pinning either mechanism.

The regression took three attempts to make honest

Each of the first two passed against known-broken code, which is worth recording because the failure mode is silent:

  1. The spy rewrote the worktree manifest — which a committed-tree scan never opens, so the mutation was a no-op. It now rewrites the manifest the loader was actually handed.
  2. The base-tree scan absorbed the single rewrite before the head scan ever parsed. Fixed with --no-base.
  3. The committed variant still passed when I reverted only the manifest_text wiring, because my own new early snapshot read masked the second parse. It fails only with that read removed as well — which is how I verified it.

Rebase

main moved to #340 (unreadable-diff classification) while I was working — 53 files, 5029 lines, heavily in cli/verify/. It rebased with no conflicts at all, which given the overlap is exactly when a clean merge deserves the least trust. Verified rather than assumed: both changesets present by marker (head_snapshot, _bind_changed_files, head_manifest_text, plan_snapshot_token, evaluated_snapshot on this side; diff_status on #340's), the diff against main is only my 8 files, both CHANGELOG entries survived, and the full suite is green on the new base.

ruff and compileall clean.

Behaviour changes accumulated across this PR

  1. verification prepare reads inputs, so it fails on a manifest whose inputs cannot load, and now also on a manifest rewritten mid-run. Previously it emitted a plan. Same conditions under which verify fails.
  2. prepare routes its errors — exit 2/3 with the agent-mode next_action/next_actions envelope — instead of printing a traceback.
  3. The out-of-root declared-path rejection applies only to the no-snapshot fallback.
  4. input_set_id / request_id values move. No schema change.

Item 1 is the one I would still like your explicit call on before merge: it is the only change here that can turn a previously-succeeding prepare into a failure.

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional error-contract regression from the latest review is attached inline.

packet_enabled=None,
packet_formats=None,
baseline_mode="new-findings",
manifest_text=read_static_input_text(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the missing-manifest config route. Reading the manifest here means FileNotFoundError is caught by the surrounding except (OSError, ValueError) and converted into InputParseError. I reproduced verification prepare in a clean Git repository without shipgate.yaml: it now exits 3 with error=input_parse_error, describes the config as an input that changed, and recommends generic review instead of emitting the established config_error/exit-2 missing-manifest setup route. This breaks agent callers that branch on the published error contract. Please preserve the loader's missing-config semantics—e.g. map a missing config_path to ConfigError before snapshot capture—while retaining InputParseError for changes after capture, and add an agent-mode regression for the error kind, exit code, and next action.

@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Confirmed and fixed in b95cef8e. This is a regression from my own manifest fix, not a pre-existing gap.

Reading the manifest through the snapshot put its FileNotFoundError inside the capture block, where the broad except (OSError, ValueError) recast it as input_parse_error/exit 3 — reporting that the config "changed while it was being read" when it simply is not there.

Two changes:

  1. The absent case is resolved before capture, delegating to load_yaml_file so the message and its init hint stay in one place rather than being duplicated at a second site.

  2. The ConfigError handler stops guessing. It now routes through the shared diagnostic catalog (_diagnose_config_error + top_next_actions) — the same path scan and doctor use. The hand-rolled handler I added earlier in this branch always emitted an "edit the manifest" action, which is simply wrong for a manifest that does not exist. The catalog already distinguishes absent / present-but-rejected / unresolved-adapter, so prepare now answers exactly as scan does, including the rank-1 verify --preview --json setup command.

Regressions

  • test_prepare_keeps_the_missing_manifest_setup_route — agent-mode: asserts config_error, exit 2, the init hint in the message, and next_actions[0] being the --preview command. Without the guard it reproduces the reported exit 3 / input_parse_error verbatim.
  • test_prepare_routes_an_unparseable_manifest_to_the_edit_action — guards the other half, so a future "fix" cannot collapse both shapes into one answer and still pass.

Full suite green, ruff and compileall clean, branch current with main.

A pattern worth naming

This is the third defect in a row with the same shape: I made a read snapshot-aware without checking what the surrounding error handling then did with the new exception, or what the surrounding blob helpers did with a path the snapshot now contains. Making a read identity-bound changes its failure modes and its absence semantics, and both need re-examining at every call site, not just the one being edited. I have stopped treating "the read now goes through the snapshot" as a local change.

Behaviour changes accumulated across this PR

  1. verification prepare reads inputs, so it fails on a manifest whose inputs cannot load, and on a manifest rewritten mid-run. Previously it emitted a plan. Same conditions under which verify fails.
  2. prepare routes its errors through the shared catalog — config_error/exit 2 for setup and edit shapes, input_parse_error/exit 3 for an input that moved — instead of printing a traceback.
  3. The out-of-root declared-path rejection applies only to the no-snapshot fallback.
  4. input_set_id / request_id values move. No schema change.

Item 1 remains the only one that can turn a previously-succeeding prepare into a failure, and it is still the call I would like from you before merge.

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional configuration-routing finding from the latest review is attached inline.

Comment thread src/agents_shipgate/cli/verification.py Outdated
actions = top_next_actions(
_diagnose_config_error(
config=str(config),
workspace=workspace,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Classify the exact config that failed. Passing workspace here makes _diagnose_config_error() ignore the explicit config and recursively discover any shipgate.yaml under the workspace. I reproduced verification prepare --workspace . --config absent-review-manifest.yaml: it emitted config_error/exit 2 but told the caller to edit the unrelated valid benchmark/perf/scenarios/large/shipgate.yaml. Monorepos and custom --config callers may therefore modify the wrong file. Please diagnose the resolved requested config with workspace=None (or give the explicit candidate precedence), and add a regression containing a missing requested manifest plus another valid manifest.

pengfei-threemoonslab and others added 7 commits August 7, 2026 22:05
`input_set_id` is the identity `verification-plan.json`,
`verification-unit-result.json`, `verify-run.json`, the terminal receipt,
and attestations all rest on, and its whole claim is that two runs sharing
it read the same bytes. Two producers broke that claim.

The manifest-derived branch of `build_verification_plan` walked only
`tool_sources`, so `openai_api.prompt_files` — and every other framework
block that names paths — never became a plan blob. Rewriting a prompt to
say refunds need no approval left `input_set_id` byte-identical.

Worse, the observed branch was inert on the committed-tree path.
`verify --base X --head Y` scans an archived copy of the head tree while
the static-input snapshot is bound to the worktree, so it recorded no
adapter reads and `active_snapshot.paths()` returned `[]` — not `None`,
so plan construction honoured an empty capture instead of falling back.
On the CI path, where the receipt is the artifact anyone downstream
actually trusts, no declared input reached the request identity at all,
not even the MCP exports and OpenAPI specs the fallback would have caught.

The two modes now agree. A committed-tree run enumerates the manifest's
declared inputs against the tree it actually scanned, and both branches
share one exclusion set so an input already hashed as a changed file is
not hashed twice. `verify` against a worktree is unchanged — read-boundary
capture already covered it, which is why the regression tests drive
`verification prepare` and the committed-tree path instead.

The declared-path table is derived from the manifest models rather than
hand-kept, so a new artifact list on an existing block, or a whole new
framework block, is covered without editing the enumeration.

One new way to fail: a declared input resolving outside the verification
input root cannot be hashed portably, so it is rejected rather than
dropped. `resolve_input_path` already rejected the same declaration the
moment an adapter read it, so this only reaches manifests naming an
out-of-root path nothing loads yet. It routes as an input error (exit 3)
with the path named; `verification prepare` routes input errors at all
now, instead of printing a traceback.

No schema changes: `plan.inputs.tool_sources` gains entries, not fields.
Existing `input_set_id` and `request_id` values do move for manifests that
declare framework inputs or that were verified with `--head`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rations

Review follow-up on #332.

Enumerating the manifest reaches only what the manifest names. An input
discovered while parsing something it names — a Google ADK `McpToolset`
inventory, an OpenAPI spec constructed inside `agent.py`, a sub-agent
config — is invisible to that walk, so two trees whose adapters read
different bytes still shared an `input_set_id`. Reproduced on the ADK
sample: the plan bound `agent.py`, the eval set, and the function
inventory, but not `inventories/mcp-tools.json`, and a trailing newline
on it left identity byte-identical.

Both remaining producers now observe the read boundary instead. A
committed-tree `verify` is snapshotted against the archived tree it
scans, which the worktree-bound snapshot could never see, and
`verification prepare` loads sources — statically, deciding nothing — to
record what they open. Committed-tree and worktree runs of the same tree
now bind the same set, asserted as an invariant rather than per-file.
The declared-path enumeration survives only as the fallback for a plan
built with no snapshot at all.

A committed-tree run therefore has two snapshots alive, and each external
input must belong to exactly one of them. The first attempt let the
archived scan read the baseline and policy packs through no snapshot at
all, silently defeating the tamper check that
`test_archived_verify_rejects_external_baseline_change_after_scan`
exists to enforce; the second let both watch the same directory, and the
second re-validation then failed on a change the first legitimately
allowed. The worktree snapshot now binds them before the archived scan
starts, which also widens the guarded window: it begins before the scan
rather than at the scan's first read.

`verification prepare` reads inputs now, so it fails on a manifest whose
inputs cannot be loaded — the same condition under which `verify` fails,
and exactly when a prepared plan could not honestly claim an input set.
Its errors carry the agent-mode envelope (`input_parse_error` /
`config_error` with `next_action` and `next_actions`, per
docs/errors.json) instead of prose alone.

With capture in place, a declared path that no adapter opens is correctly
not an input, so the out-of-root rejection now applies only to the
declared fallback; its test moved to target that path directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review follow-up on #332.

Capture recorded which paths the adapters opened and then released the
snapshot, leaving `build_verification_plan` to reopen those files to hash
them. The path list and the blob hashes therefore came from two different
instants. A file rewritten in between is attested at its new content while
`tool_sources` still lists what the old content pointed at — so the plan
can bind a pre-scan `inventories/mcp-tools.json`, omit the inventory the
post-scan `agent.py` actually names, and hash `agent.py` at bytes the
report never evaluated. Reproduced on `prepare`: captured `agent.py` was
sha256:7465..., the emitted plan recorded sha256:040c....

My earlier note here claimed the reset was safe because the archived tree
is a private temporary directory. That reasoned about who can write the
files and missed the actual defect, which is that the two halves of the
plan are taken at different times regardless of the threat model.

Plan construction now runs under the finalized snapshot on both paths:
`verify` re-activates the head snapshot around the plan build, and
`prepare`'s capture became a context manager that holds the snapshot open
for it. Blobs are hashed from captured bytes and never reopened.

That exposed a second-order requirement. `_blobs` skips a path the
snapshot contains but never read, so binding the snapshot without also
binding the changed files silently dropped every changed file no adapter
opens — verified: `changed_files` goes to empty. The worktree path already
preloaded changed files for exactly this reason; that block is now
`_bind_changed_files` and is applied to the archived tree and to
`prepare` as well.

Both regressions fail against the unfixed code, checked by reverting each
activation independently, and the committed-tree one also asserts an
unrelated changed file survives capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running plan construction under the finalized snapshot turned binding
from an optimization into an obligation. Under an active snapshot both
`_blobs` and `_optional_blob` read a path that is *contained but never
read* as absent, so an input nobody bound does not merely go unhashed —
it disappears from the plan. I named that trap for `changed_files`, fixed
that one case, and did not sweep the rest, so `--baseline` and
`--diff-from` silently dropped: two preparations of one tree with
different A/B files emitted `baseline: null`, `diff_from: null`, and the
same `input_set_id`.

Every input `build_verification_plan` hashes is now bound before the
snapshot is sealed — the adapters' own reads, the changed files, and the
explicit baseline, comparison report, and policy packs. Audited rather
than patched at the reported site: `_blobs` and `_optional_blob` are the
two helpers that drop unread paths, and every call to them inside the
plan builder is covered. `build_blob` and `sha256_file` gate on `has()`
with a direct-read fallback, so they cannot drop.

The comparison report is never mapped into the archived tree, so on a
committed-tree preparation it is bound as an external input; without that
the snapshot refuses to read it at all.

The archive directory is also resolved once it exists. macOS reaches it
through /var while every derived path resolves to /private/var, and the
snapshot matches lexically, so two spellings make `contains()` false for
inputs plainly inside the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`load_manifest_with_positions` reads the manifest twice — a direct
`Path.read_text` to build the model, then through the snapshot to build
the position index. A rewrite between the two lets the adapters follow
one manifest while the plan's config blob attests to another, so a
receipt could name an entrypoint the scan never opened.

The worktree path has always dodged this by passing its captured
`worktree_manifest_text`. The committed-tree path passed None, and the
`prepare` capture added earlier in this branch passed nothing either.
Both now read the manifest once through the active snapshot and hand
those exact bytes to `_prepare_scan` / `run_scan`.

The archived tree is also resolved at creation rather than at each use,
so the snapshot's lexical matching sees one spelling: on macOS the
temporary directory is reached through /var while every adapter resolves
its base directory to /private/var.

Two correct outcomes follow, and the regression asserts the invariant
rather than either mechanism: `prepare` now fails closed because
`finish()` catches the rewrite, while committed-tree `verify` emits a
self-consistent plan. What must never happen is succeeding with a plan
that disagrees with the manifest the scan followed.

Getting that regression honest took three attempts, each of which passed
against known-broken code: the spy first rewrote the worktree manifest,
which a committed-tree scan never opens; then the base-tree scan absorbed
the single rewrite before the head scan parsed; and the committed variant
still passed when only the `manifest_text` wiring was reverted, because
the new early snapshot read masked it. It fails only with that read
removed as well.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading the manifest through the snapshot put its FileNotFoundError inside
the capture block, where a broad `except (OSError, ValueError)` recast it
as `input_parse_error` / exit 3 — telling an agent the config changed
while it was being read when it simply is not there, and offering generic
review instead of setup. A caller branching on the published error
contract is stranded by that.

The absent case is now resolved before capture, delegating to
`load_yaml_file` so the message and its `init` hint stay in one place.

The `ConfigError` handler also stops guessing. It routes through the
shared diagnostic catalog (`_diagnose_config_error` + `top_next_actions`),
the same path `scan` and `doctor` use, so an absent manifest gets the
setup route, an unparseable one gets the edit route, and an unresolved
adapter gets its own. The hand-rolled handler added earlier in this branch
always emitted an "edit the manifest" action, which is wrong for a
manifest that does not exist.

Two agent-mode regressions: the missing-manifest kind, exit code, and
next action, and an unparseable manifest still reaching the edit route
rather than being flattened into the same answer. Without the guard the
first reproduces the reported exit 3 / input_parse_error verbatim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Routing prepare's ConfigError through the shared diagnostic catalog last
commit copied `scan`'s call shape without checking what the arguments
mean. `_candidate_manifest_paths` short-circuits on `workspace is not
None` and recursively discovers every `shipgate.yaml` beneath it,
ignoring the explicit `--config` entirely, so the first discovered file
that happens to parse became the subject of the recovery action.

Reproduced in this repository: `verification prepare --workspace .
--config absent-review-manifest.yaml` named the absent file in its
message but told the caller to edit
`benchmark/perf/scenarios/large/shipgate.yaml`, an unrelated and
perfectly valid fixture. A monorepo, or any invocation naming a config
that is not there, would be routed at someone else's manifest.

`prepare` always resolves exactly one manifest path, so there is nothing
to discover: diagnose that resolved path with `workspace=None`. `scan`
passes both arguments and is right to, because its `-c` may be absent or
a glob; the difference is what I failed to check.

The regression plants a second valid manifest at
packages/other/shipgate.yaml and asserts it appears nowhere in
`next_actions`. Reverting the fix fails it with that exact path in an
edit action.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-299-cb8160 branch from 45e2e18 to 2f1b2b7 Compare August 8, 2026 05:06
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Confirmed and fixed in 2f1b2b79. This is a flaw in the fix I made last round, not a pre-existing gap.

When I switched to the shared diagnostic catalog I copied scan's call shape without checking what the arguments mean. _candidate_manifest_paths short-circuits on workspace is not None and recursively discovers every shipgate.yaml beneath it, ignoring the explicit --config entirely, so the first discovered file that happens to parse became the subject of the recovery action.

Reproduced in this repository:

verification prepare --workspace . --config absent-review-manifest.yaml

The message correctly named the absent file; next_actions[0] was edit → benchmark/perf/scenarios/large/shipgate.yaml, an unrelated and perfectly valid fixture manifest.

prepare always resolves exactly one manifest path, so there is nothing to discover — it now diagnoses that resolved path with workspace=None. scan passes both arguments and is right to, because its -c may be absent or a glob; the difference is what I failed to check.

Both routes verified after the fix: an absent --config alongside other valid manifests gets the setup route with no mention of the unrelated file, and a missing default manifest still gets the setup route unchanged.

Regression

test_prepare_diagnoses_the_requested_config_not_a_discovered_one plants a second valid manifest at packages/other/shipgate.yaml and asserts it appears nowhere in next_actions. Reverting the fix fails it with that exact path in an edit action.

One caveat on my local verification

main picked up the pytest bump to >=9.1.1 (#301) while I was working, and I rebased onto it. My local interpreter still has pytest 9.0.3, so my "full suite green" for this round was run on 9.0.3, not the version CI installs. The suite is green there and ruff/compileall are clean, but CI's test job is the authoritative signal for 9.1.1 and I have not independently reproduced it. Flagging rather than letting the claim stand unqualified.

Behaviour changes accumulated across this PR

  1. verification prepare reads inputs, so it fails on a manifest whose inputs cannot load, and on a manifest rewritten mid-run. Previously it emitted a plan. Same conditions under which verify fails.
  2. prepare routes its errors through the shared catalog — config_error/exit 2 for setup and edit shapes, input_parse_error/exit 3 for an input that moved — instead of printing a traceback.
  3. The out-of-root declared-path rejection applies only to the no-snapshot fallback.
  4. input_set_id / request_id values move. No schema change.

Item 1 remains the only one that can turn a previously-succeeding prepare into a failure, and it is still the call I would like from you before merge.

@pengfei-threemoonslab
pengfei-threemoonslab merged commit c07ba35 into main Aug 8, 2026
4 checks passed
@pengfei-threemoonslab
pengfei-threemoonslab deleted the claude/github-issue-299-cb8160 branch August 8, 2026 07:33
pengfei-threemoonslab added a commit that referenced this pull request Aug 8, 2026
Resolves the collision with #332 (declared adapter input identity).

#332 refactored `prepare` into `_build_plan`/`_captured_inputs` and
extracted `_bind_changed_files`, and made `_worktree_overlay` treat a path
the static snapshot contains but never read as absent. This branch adds a
second, deliberately disjoint path set — `worktree_overlay_paths` is
HEAD-relative while `changed_files` is merge-base-relative — so a canceled
path exists only in the former.

Binding just `changed_files` therefore recorded a present canceled file as
`deleted`, and `verification worker` failed with "worker worktree overlay
does not match the plan". Both binding sites now bind the union.
pengfei-threemoonslab added a commit that referenced this pull request Aug 8, 2026
The merge with #332 exposed a semantic conflict the suite could not see.
#332 makes the static input snapshot report a path it contains but never
read as absent, and this branch's overlay set is HEAD-relative while the
change set is merge-base-relative — so a cancelled-but-present path is
bound by neither unless both binding sites take the union.

The existing cancellation test cancels by deleting the file, so the
producer records "deleted" either way and the corruption is invisible.
This one cancels by restoring merge-base content, then calls
_validate_git_subject: it fails with "worker worktree overlay does not
match the plan" if either binding site drops back to changed_files.
pengfei-threemoonslab added a commit that referenced this pull request Aug 9, 2026
* Normalize effective worktree verification diffs

* Address verification diff review feedback

* test: cover a cancelled overlay path that is still present

The merge with #332 exposed a semantic conflict the suite could not see.
#332 makes the static input snapshot report a path it contains but never
read as absent, and this branch's overlay set is HEAD-relative while the
change set is merge-base-relative — so a cancelled-but-present path is
bound by neither unless both binding sites take the union.

The existing cancellation test cancels by deleting the file, so the
producer records "deleted" either way and the corruption is invisible.
This one cancels by restoring merge-base content, then calls
_validate_git_subject: it fails with "worker worktree overlay does not
match the plan" if either binding site drops back to changed_files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

input_set_id omits adapter-read inputs (prompt files, framework sources)

1 participant