Skip to content

fix(collector): Polymarket tape recorder size cap + uploader throughput/reliability #655

Description

@proerror77

Background / 背景

Production incident 2026-08-03 (runtime tracking: #533): after tick-level recording (#548), Polymarket market tapes grew to 20–25 GB/h. The upload service (polymarket-market-tape-upload.service) was found stopped for 19 h; backlog hit 145 GB and /data nearly filled (4.1 GB free), one hour from a recorder data hole. Drain required manual dual-lane relay (ECS + 5 parallel ACK jobs).

Root causes that remain unfixed in code:

  1. Uploader throughput (rust_hft/tools/collector/src/polymarket_upload.rs):

    • zstd invoked as zstd -q -T1 -3 (single-threaded) — compressing a 20 GB tape takes minutes and previously hit ZSTD_TIMEOUT_SECONDS=300 under CPUQuota=30%.
    • aliyun ossutil cp invoked without --parallel/--part-size — measured ~50 MiB/s vs ~100+ MiB/s with --parallel 12 --part-size 32Mi on the same host (intranet baseline ~1 Gbps).
    • Host binary is 54710b06; upstream origin/main additionally has fix(collector): bound async Polymarket OSS uploads #620 bounded concurrent uploads (undeployed).
  2. Recorder has no tape size cap (rust_hft/tools/collector/src/polymarket_raw.rs): rotation is UTC-hour only, so one tape can reach 20–25 GB. Per-tape processing needs ~1.6× tape size of transient disk (hour-split copy + .zst + verify download), which is what deadlocks a 196 GB spool disk once a few tapes backlog.

  3. No self-healing: nothing restarts a dead upload timer/service; the 19 h stop was discovered by accident.

Scope / 范围

  • PR A (uploader throughput): configurable zstd threads (default auto) and ossutil --parallel/--part-size (env-tunable, sane defaults) for upload and download invocations in polymarket_upload.rs, with unit tests.
  • PR B (recorder size cap): rotate the active tape when it exceeds POLYMARKET_TAPE_MAX_BYTES (default ~4 GiB) in addition to the UTC-hour rotation, reusing the existing fail-closed rotation path, with tests.
  • Watchdog (deploy config): a small systemd timer that restarts polymarket-market-tape-upload.timer if it is not active and logs spool/disk state.
  • After merge: publish image, deliver binary to monday-trade-data-26, cut over with controller amendment, verify end-to-end on live tapes.

Out of scope (follow-ups): reference-collector crash-loop + its 21 GB backlog (#539 line); canonical Parquet zone; per-collector spool isolation.

Acceptance / 验收

  • cargo test -p hft-collector --locked green on both PRs; CI gates pass.
  • Live readback: new binary --version on host; a newly rotated tape uploads end-to-end (triplet + readback) without manual intervention; watchdog timer active; upload keeps pace with tick-level inbound for ≥2 h.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestready-for-agentFully specified and safe for an autonomous agent

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions