fix(pm-gate): real-market preflight budget too small for tick-level segments

[proerror77]

Summary

REAL_MARKET_PREFLIGHT_BUDGET_SECONDS = 1200 in polymarket-raw-ops-shadow-gate.sh was sized when the largest observed hourly market segment was ~109MiB. Tick-level recording (quote_sample_ms = 0) produces 7.7–18GB hourly segments, and the preflight's full-file passes (jq quote count + recorded-hours scan + candidate validate/compress/upload/readback) exceed 1200s, so the worker is SIGKILLed and the gate dies with candidate rejected a real production closed market segment before shadow startup — a timeout, not a validation rejection.

Observed 2026-08-02, gate invocation 7c44f5f1b00041b48f21315e4bf7ed2b (candidate 5aa208a1): 20min10s wall ≈ exactly the 1200s budget, 10m58s CPU, no preflight upload reached OSS.

Context / workaround used

Gate was unblocked operationally by parking the multi-GB segments (/data/monday/spool/polymarket-parked/) and producing a small recorder-closed segment (~1.4GB, two bounded recorder restarts) so the preflight fits the budget. The parked segments return to the spool after cutover; the fixed uploader (#584) retries them.

Change contract (proposal)

The market preflight must prove candidate upload correctness on a real segment without scaling cost with tick volume. Options:

• Bound the preflight scan (e.g., validate a deterministic head+tail window of the segment plus full-file cheap checks) — strongest fix, keeps coverage meaningful;
• Or pick the smallest closed segment instead of the latest;
• Or scale the budget from the picked segment's size with an explicit upper bound (weakest — a 20GB hour still costs 30+ min of gate time).

Any option needs a counterexample test: a >1200s-budget-sized synthetic segment must not make the preflight exceed its deadline.

Out of scope

• The upload validator jitter tolerance itself (fix(pm-upload): tolerate bounded per-token quote source-time jitter for tick-level tapes #584, merged).
• Backlog reprocessing mechanics.

Acceptance

• Gate passes with a multi-GB latest closed segment in the spool, without manual parking.
• Counterexample test for the budget/scan bound.

[proerror77]

Fix design review (2026-08-07) — recommended approach after implementation analysis.

Evaluated options

1. Pick smallest segment — REJECTED. Does not satisfy the acceptance test "a >1200s-budget-sized synthetic segment must not make preflight exceed its deadline": if the spool has only one multi-GB segment, "smallest" == the huge one, and preflight still times out. Also changes preflight's validation semantics (no longer validates the latest closed segment).

2. Scale budget by segment size — REJECTED (as issue notes): a 20GB hour still costs 30+ min of gate time.

3. Bound the preflight scan (head+tail window + cheap full-file checks) — RECOMMENDED. Keeps coverage meaningful, cost does not scale with tick volume, satisfies the counterexample test.

Recommended implementation (option 3)

In real_market_segment_preflight (deployment/aliyun/polymarket-raw-ops-shadow-gate.sh ~L925-934), the two full-file jq passes are the cost driver:

• jq -c 'select(.update.kind == "quote")' → wc -l (quote count)
• jq -r '.recorded_at ...' | sort -u | wc -l (recorded-hours check)

Change contract:

• Head+tail window: validate a deterministic window (e.g. first and last N records) for quote presence and recorded_at hour consistency.
• Full-file cheap check: keep a jq -e empty or jq -e structural pass that costs O(n) but is memory-light and far faster than full select+wc; or use head/tail sampling with an explicit record cap.
• Evidence: the preflight JSON currently records source_quote_records as an exact integer. This must change to a sampled/capped value with an explicit field rename or annotation (e.g. source_quote_records_sampled) so evidence stays honest (per repo "never fabricate completeness" rule).

Counterexample test required (acceptance)

A synthetic segment large enough that the current full-scan would exceed REAL_MARKET_PREFLIGHT_BUDGET_SECONDS must complete preflight within the deadline under the bounded scan. The existing test-polymarket-raw-ops-control-plane.sh preflight harness (L799-1188) is the place to add this case.

Ownership note

This is a production deployment-script change (runtime/deploy path). Per repo rules it needs a named controller, the exact counterexample test, and a controlled rollout. Recommended as a dedicated worktree/PR rather than folded into other work.

[proerror77]

Implementation status: design approved (see fix design review above). Ready for implementation in a dedicated worktree/PR. Implementation should follow the recommended approach and include the counterexample test named in the acceptance criteria.