polymarket parity validator cannot validate high-rate tapes: stable-read race on the live baseline spool

[proerror77]

Current behavior

verify-shadow-parity (rust_hft/tools/collector/src/polymarket_parity.rs:1245) requires every tape in both spools — including the still-running baseline lane's active tape — to remain byte- and mtime-stable across two full sequential parses. FileFingerprint equality (device/inode/size/mtime-nsec, polymarket_parity.rs:79-98) is checked before, during, and after each pass in stream_stable_rows (:236-289); any append or rotation returns "changed" and load_rows (:345-419) restarts from scratch, with only 5 retries at 20 ms before it bails with spool changed while reading parity window (:418).

The shadow gate stops and finalizes only the shadow lane (deployment/aliyun/polymarket-raw-ops-shadow-gate.sh:1901-1916), then runs the verifier against the live baseline spool (:1922-1926), while LEGACY_RUNTIME_STABILITY_REQUIRED=true (:25) requires that baseline to stay active and collecting throughout. On high-rate tapes the baseline appends (or rotates) during a pass, the retry budget exhausts, and the gate produces no parity evidence at all — byte/field/dedupe/settlement/rotation parity are all unvalidated and the promotion gate fails closed.

Amplifiers, same file: read cost per attempt is O(total spool bytes), not O(comparison window) — tape_paths (:162-189) enumerates every segment and both passes parse every row (:358, :385), so pass duration grows with spool retention and raises the per-attempt race probability; and all in-window rows are retained in memory with full Value clones (:359-361, :410, :668), so peak memory scales with tape rate on a 7.75 GiB host.

Focused reproduction

Code-level, deterministic: a fixture where a writer appends valid rows to the legacy spool's active tape every ~10 ms while compare() runs makes the fingerprint check fail every attempt and reproduces the load_rows bail within 5 attempts. Per-attempt failure probability approaches 1 as the mean inter-append interval drops below the two-pass read duration; no retry-count or sleep tuning changes that asymptote. Not yet observed against a live high-rate tape — this is a structural analysis with a synthetic reproducer, stated plainly in the known-issue doc.

Known-issue write-up: docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md.

Expected behavior

Parity validation completes on live high-rate tapes with read cost bounded by the comparison window (not total spool size or writer quiescence) and peak memory bounded independently of the in-window trade count — while preserving every fail-closed guarantee (sequence gaps, truncation, rename/indirection, schema violations, duplicate identities still fail).

Safety and scope boundary

• Fail-closed semantics are the contract: the fix must not weaken any existing gate check in deployment/aliyun/polymarket-shadow-gate-policy.jq.
• Evidence schema compatibility: monday.polymarket_shadow_parity.v1 metrics consumed by the gate policy stay compatible, or schema and policy are versioned together in the same change.
• Runtime/gate recover logic is out of scope (owned by a separate follow-up).
• One PR = one behavior: validator read/windowing change only; no collector or gate-script behavior changes beyond what the validator interface requires.