investigate: preflight-scan change to shadow-gate triggers snapshot test failure (cross-impact on #586)

[proerror77]

Problem

Issue #586 (PR #730, bounded preflight scan) is blocked: its Rust Workspace check fails at the snapshot test (exercise_bootstrap_snapshot) with snapshot rejected: ...digest changed before rollback manifest. Investigation shows a cross-impact: the #586 preflight-scan change to polymarket-raw-ops-shadow-gate.sh triggers the snapshot test to fail, even though the two are code-independent.

What is known (all experiments)

Experiment	Result	Conclusion
main (after #732 sync fix) Monorepo CI gate	PASS	snapshot is green on main
#586 branch (preflight change + counterexample)	FAIL	#586 change triggers snapshot failure
#586 branch without counterexample	still FAIL	counterexample is NOT the cause
Local GNU-coreutils reproduction (full overrides)	PASS	logic is correct in isolation
#731 sync fix (PR #732)	merged but ineffective	not the root cause

Established facts

• snapshot rejected comes from polymarket-raw-ops-cutover.sh:1080 (snapshotted Rust bootstrap binary digest changed before rollback manifest), i.e. after snapshot_legacy copies ACTIVE_BINARY into the rollback dir and sha256sum --check --stricts it against baseline_release_sha.
• On CI the digest genuinely mismatches (sha256sum: WARNING: 1 computed checksum did NOT match); locally it matches.
• The failing case is exercise_bootstrap_snapshot (test harness), whose install()/stat() overrides interact with snapshot_legacy's copying of /etc/systemd/system/* and the bootstrap binary.
• Removing the counterexample from the test script does not change the outcome, so the trigger is the preflight-scan change to polymarket-raw-ops-shadow-gate.sh (or its interaction with the test harness's sed extraction of preflight functions).

Hypotheses to investigate

1. Preflight sed-extraction interaction: the test script sed-extracts real_market_segment_preflight + PREFLIGHT_SCAN_WINDOW_RECORDS (L799-808). The preflight change may alter what is extracted, changing shell state that the later snapshot test relies on.
2. stat() override over-broad: test stat() { printf '0755\n'; } returns 0755 for both -c %a (mode) and -c %u/%U (owner) queries. If the override is not scoped correctly for the bootstrap path, secure_regular_file's root-owner check breaks.
3. install() override redirect: install -m "$mode" "/etc/systemd/system/$asset" must redirect to $systemd_fixture; if the redirect misses for a new path, the install fails and the copy target differs.
4. CI environment/timing: main passes but fix(pm-gate): real-market preflight budget too small for tick-level segments #586 fails consistently; the preflight scan change may alter test ordering or resource usage such that the snapshot runs under different conditions.

Recommended approach

• Dedicated worktree/branch on main; run deployment/aliyun/test-polymarket-raw-ops-control-plane.sh on CI (Linux) with only the preflight change, then bisect which part (sed extraction line, PREFLIGHT_SCAN_WINDOW_RECORDS constant, or the shadow-gate scan body) triggers the snapshot failure.
• Add targeted debug to exercise_bootstrap_snapshot (print baseline_sha, rollback-bin sha, ACTIVE_BINARY sha at failure) — the existing SNAPSHOT_DEBUG instrumentation showed copied-binary-drift digest mismatch is expected, so focus on absent-control.

Blocks

• PR fix(pm-gate): bound real-market preflight scan for tick-level segments #730 (issue fix(pm-gate): real-market preflight budget too small for tick-level segments #586 bounded preflight scan)
• Issue fix(test): polymorpharket raw-ops control-plane snapshot test fails on CI when Rust Workspace job runs #731 (snapshot test flake, reopened)

[proerror77]

Resolution (2026-08-07): ROOT CAUSE FOUND — the snapshot failure is a PRE-EXISTING test defect, NOT a cross-impact from #586.

Decisive evidence

Running the test locally (with GNU coreutils wrappers for macOS) on a completely clean baseline — main polymarket-raw-ops-shadow-gate.sh + main test-polymarket-raw-ops-control-plane.sh, zero #586 changes — still fails at the snapshot test:

snapshot rejected: snapshotted Rust bootstrap binary digest changed before rollback manifest

Why main's CI appeared to pass

• main's Monorepo CI gate Rust Workspace job is skipped by CI scope (it only runs when the change touches collector/deployment paths).
• So the snapshot test was never validated by main CI.
• fix(pm-gate): real-market preflight budget too small for tick-level segments #586's deployment-script change triggered the Rust Workspace job to run, which first exposed this pre-existing defect.

Conclusion

• fix(pm-gate): real-market preflight budget too small for tick-level segments #586 (bounded preflight scan) is not the cause; its own logic is verified (counterexample passes).
• The snapshot test has a pre-existing, hidden defect that only manifests when the Rust Workspace job actually runs.
• This is tracked in fix(test): polymorpharket raw-ops control-plane snapshot test fails on CI when Rust Workspace job runs #731 (reopened). It blocks any PR that triggers Rust Workspace (i.e. any deployment-script PR) until fixed.

[proerror77]

Investigation concluded (2026-08-07) — the cross-impact was resolved as part of PR #730.

Findings

1. The snapshot failure was a PRE-EXISTING test defect (confirmed: clean main baseline fails locally too). Main CI never ran it because the Rust Workspace job is scope-skipped.
2. Root cause: die() in exercise_bootstrap_snapshot uses exit 1, which under the harness's set -euo pipefail + subshell-function context terminated the whole script when the die fired inside the if-condition (the copied-binary-drift case).
3. Fix (in fix(pm-gate): bound real-market preflight scan for tick-level segments #730): run snapshot_legacy in a nested subshell so the die's exit 1 confines to that subshell; the caller's if observes the status normally. Also added source_scan_bounded:true to the preflight fixture required by the policy change.

Outcome

• fix(pm-gate): real-market preflight budget too small for tick-level segments #586 (bounded preflight scan) merged in PR fix(pm-gate): bound real-market preflight scan for tick-level segments #730; Rust Workspace passes.
• The earlier hypothesis of a "cross-impact from the preflight-scan change" was refuted — the change merely exposed a hidden pre-existing defect.

Closing as resolved.