bug(polymarket-release): gate recover refuses failed-state contained units, no governed remediation

[proerror77]

Parent

None — follow-up to #637 (recover action contract). Full analysis: docs/reports/2026-08-08-polymarket-raw-ops-gate-recover-known-issue.md.

Current behavior

recover in deployment/aliyun/polymarket-raw-ops-gate-control.sh (recover_gate, :727-738) admits a recovery gate only when the contained baseline collector reports ActiveState == "inactive" (:676-679) and all four uploader units/timers report exactly inactive (verify_recovery_uploaders_stopped, :717-725). A unit in failed state — no managed process, restart budget exhausted or Restart=no, i.e. fully contained — is refused (recovery requires the direct bootstrap baseline to be stopped / recovery requires inactive uploader/timer). failed is the modal post-incident state (crash-loop exhaustion, failed oneshot uploaders), i.e. exactly the conditions the recover action was scoped to in #637.

Downstream consumers bind the recorded snapshot, not the inactive constant: gate verify_contained_recovery_baseline compares live state to the snapshot (polymarket-raw-ops-shadow-gate.sh:529-531); the "inactive" literal appears only in binding predicates (polymarket-raw-ops-shadow-gate.sh:501, polymarket-shadow-gate-policy.jq:80, polymarket-raw-ops-cutover.sh:964). The refusal is therefore an admission-time artifact, not a safety requirement.

Secondary gaps: admission preconditions run before CONTROL_LOCK is taken (:764-765) and refused admissions leave no durable evidence (stderr only); baseline identity is bound from systemctl-loaded values only, never on-disk fragment bytes (contrast host-rust-lob-restore.sh:444-456).

Focused reproduction

On the collector host (or in the harness, test-polymarket-raw-ops-control-plane.sh:434-488, which currently never exercises failed):

1. Contain the baseline by letting it fail (e.g. crash-loop with exhausted restart budget), leaving polymarket-reference-collector.service in ActiveState=failed, MainPID=0.
2. Run polymarket-raw-ops-gate-control.sh recover <candidate-binary> <sha256> <source-revision> <gamma-probe.json> with a fresh, valid probe.
3. Observe refusal: recovery requires the direct bootstrap baseline to be stopped. Same for any failed uploader oneshot: recovery requires inactive uploader/timer.

No governed remediation exists; the only paths forward are unrecorded manual systemctl reset-failed/stop outside the control lock, or manual stop/start of the production collector to force inactive (ungoverned production runtime transition, no health verification).

Expected behavior

recover admits any verifiably contained baseline/uploader (MainPID == 0, not active, exact identity binding unchanged), including failed units; units observed failed are brought to inactive via a governed systemctl reset-failed step inside the control lock (the host-rust-lob-restore.sh:472 precedent), the post-reset state is re-read into the baseline snapshot, and every admission (accepted or refused) writes an immutable evidence record with exact identities and reason. All existing fail-closed checks (probe freshness, identity binding, active-unit refusal) stay unchanged.

Safety and scope boundary

• Trust domains: Aliyun collector host monday-trade-data-26 (ap-northeast-1) systemd control plane and gate evidence chain. No runtime mutation is authorized by this issue; any fix's runtime application remains a separately authorized cutover.
• Must remain unchanged: probe freshness/binding (polymarket-raw-ops-gate-control.sh:640-671), MainPID == 0 containment, exact identity binding, the parity validator (owned by polymarket parity validator cannot validate high-rate tapes: stable-read race on the live baseline spool #747 and its own write-up), and all existing refusal behavior for active/activating/deactivating units.
• Relation to precedent: the fix should meet the host-rust-lob-restore.sh bar — quiescence via is-active (:425-428), governed reset-failed (:472), lock-first serialization (:517-526), and failure evidence (:202-241, :306-322).

Acceptance criteria

• Harness: failed baseline and each failed uploader admitted after governed reset-failed; recorded snapshot reads inactive/MainPID 0; gate binding, gate policy, and cutover recovery binding pass unchanged end-to-end.
• Regression: active/activating/deactivating baselines, active uploaders, and all existing refusal cases still refuse; baseline-mutation assertion (test-polymarket-raw-ops-control-plane.sh:481-488) preserved and extended to prove reset-failed occurs only inside the control lock.
• Admission evidence: admitted and refused recover invocations write immutable records, asserted in the harness.
• test-polymarket-raw-ops-control-plane.sh passes; shellcheck/bash -n clean; git diff --check clean.

Blocked by

None - can start immediately.

Out of scope

Parity validator changes (#747), production runtime mutation, uploader behavior changes, and any weakening of existing fail-closed checks.