You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Production incident 2026-08-03 (runtime tracking: #533): after tick-level recording (#548), Polymarket market tapes grew to 20–25 GB/h. The upload service (polymarket-market-tape-upload.service) was found stopped for 19 h; backlog hit 145 GB and /data nearly filled (4.1 GB free), one hour from a recorder data hole. Drain required manual dual-lane relay (ECS + 5 parallel ACK jobs).
zstd invoked as zstd -q -T1 -3 (single-threaded) — compressing a 20 GB tape takes minutes and previously hit ZSTD_TIMEOUT_SECONDS=300 under CPUQuota=30%.
aliyun ossutil cp invoked without --parallel/--part-size — measured ~50 MiB/s vs ~100+ MiB/s with --parallel 12 --part-size 32Mi on the same host (intranet baseline ~1 Gbps).
Recorder has no tape size cap (rust_hft/tools/collector/src/polymarket_raw.rs): rotation is UTC-hour only, so one tape can reach 20–25 GB. Per-tape processing needs ~1.6× tape size of transient disk (hour-split copy + .zst + verify download), which is what deadlocks a 196 GB spool disk once a few tapes backlog.
No self-healing: nothing restarts a dead upload timer/service; the 19 h stop was discovered by accident.
Scope / 范围
PR A (uploader throughput): configurable zstd threads (default auto) and ossutil --parallel/--part-size (env-tunable, sane defaults) for upload and download invocations in polymarket_upload.rs, with unit tests.
PR B (recorder size cap): rotate the active tape when it exceeds POLYMARKET_TAPE_MAX_BYTES (default ~4 GiB) in addition to the UTC-hour rotation, reusing the existing fail-closed rotation path, with tests.
Watchdog (deploy config): a small systemd timer that restarts polymarket-market-tape-upload.timer if it is not active and logs spool/disk state.
After merge: publish image, deliver binary to monday-trade-data-26, cut over with controller amendment, verify end-to-end on live tapes.
Out of scope (follow-ups): reference-collector crash-loop + its 21 GB backlog (#539 line); canonical Parquet zone; per-collector spool isolation.
Acceptance / 验收
cargo test -p hft-collector --locked green on both PRs; CI gates pass.
Live readback: new binary --version on host; a newly rotated tape uploads end-to-end (triplet + readback) without manual intervention; watchdog timer active; upload keeps pace with tick-level inbound for ≥2 h.
Background / 背景
Production incident 2026-08-03 (runtime tracking: #533): after tick-level recording (#548), Polymarket market tapes grew to 20–25 GB/h. The upload service (
polymarket-market-tape-upload.service) was found stopped for 19 h; backlog hit 145 GB and/datanearly filled (4.1 GB free), one hour from a recorder data hole. Drain required manual dual-lane relay (ECS + 5 parallel ACK jobs).Root causes that remain unfixed in code:
Uploader throughput (
rust_hft/tools/collector/src/polymarket_upload.rs):zstd -q -T1 -3(single-threaded) — compressing a 20 GB tape takes minutes and previously hitZSTD_TIMEOUT_SECONDS=300underCPUQuota=30%.aliyun ossutil cpinvoked without--parallel/--part-size— measured ~50 MiB/s vs ~100+ MiB/s with--parallel 12 --part-size 32Mion the same host (intranet baseline ~1 Gbps).54710b06; upstreamorigin/mainadditionally has fix(collector): bound async Polymarket OSS uploads #620 bounded concurrent uploads (undeployed).Recorder has no tape size cap (
rust_hft/tools/collector/src/polymarket_raw.rs): rotation is UTC-hour only, so one tape can reach 20–25 GB. Per-tape processing needs ~1.6× tape size of transient disk (hour-split copy + .zst + verify download), which is what deadlocks a 196 GB spool disk once a few tapes backlog.No self-healing: nothing restarts a dead upload timer/service; the 19 h stop was discovered by accident.
Scope / 范围
--parallel/--part-size(env-tunable, sane defaults) for upload and download invocations inpolymarket_upload.rs, with unit tests.POLYMARKET_TAPE_MAX_BYTES(default ~4 GiB) in addition to the UTC-hour rotation, reusing the existing fail-closed rotation path, with tests.polymarket-market-tape-upload.timerif it is not active and logs spool/disk state.monday-trade-data-26, cut over with controller amendment, verify end-to-end on live tapes.Out of scope (follow-ups): reference-collector crash-loop + its 21 GB backlog (#539 line); canonical Parquet zone; per-collector spool isolation.
Acceptance / 验收
cargo test -p hft-collector --lockedgreen on both PRs; CI gates pass.--versionon host; a newly rotated tape uploads end-to-end (triplet + readback) without manual intervention; watchdog timer active; upload keeps pace with tick-level inbound for ≥2 h.