fix(collector): PM tape upload rejects quote_collection_failure kind, segments stuck since 2026-07-30

[proerror77]

Change contract

polymarket-raw-ops upload must accept the quote_collection_failure record kind that new-ploy-runner legitimately writes into market tapes (empty / one-sided / non-executable books), instead of failing validation.

Evidence

Host monday-trade-data-26 (i-6we6afeqsvv8uo1ixmyo), 2026-07-31:
/data/monday/spool/polymarket/upload-status.json shows every segment since 2026-07-30T22:00 failing with unsupported update kind 'quote_collection_failure'; 14 segments pending, last_success_at=2026-07-30T22:35:06Z. Recorder emits these records per ploy-strategy-bundles/src/feed/recorded.rs (QuoteCollectionFailure "empty"/"non_executable"). Upload validator allowlist rust_hft/tools/collector/src/polymarket_upload.rs:24-34 (ALLOWED_KINDS) lacks the kind; scan bails at :976-977.

Acceptance

• Upload validator accepts quote_collection_failure records with field-level validation consistent with the recorder's schema
• Manifest quality fields reflect failure records (not counted as quotes) without weakening quote-coverage semantics
• Unit tests: tape containing quote_collection_failure passes scan/validate; malformed failure records still rejected
• After deploy, the 14 retained spool segments upload successfully (runtime evidence)

Out of scope

• Binance reference kinds on PM tape (separate issue)
• Tick-level sampling config change

Runtime control

Target: monday-trade-data-26 (ECS i-6we6afeqsvv8uo1ixmyo) polymarket-market-tape-upload.service
Candidate: current exact-main uploader binary (validator accepts quote_collection_failure since #239; regression tests merged in #543)
Controller: kimi (authorized by repo owner 2026-07-31)
Stop rule: after redeploy, upload-status.json still shows pending>0 for the 14 retained segments or new upload failures appear
Rollback: redeploy the previous uploader binary and restore the prior spool state

[proerror77]

Audit status (2026-08-01)

代码合约已交付:main 校验器自 #239 起支持 quote_collection_failure,PR #543 已合入回归测试(6 个新测试,58 个 polymarket_upload 测试绿)。剩余唯一验收项是运行时证据:将当前 uploader 二进制重新部署到 monday-trade-data-26(i-6we6afeqsvv8uo1ixmyo),确认 14 个滞留 segment 全部上传成功(/data/monday/spool/polymarket/upload-status.json pending=0、last_success_at 更新),然后按 runtime 关闭门禁记录证据并关闭。

The code contract is delivered (validator supports the kind at main since #239; regression tests merged in #543). The only outstanding acceptance item is runtime evidence: redeploy the current uploader binary to monday-trade-data-26, verify the 14 retained segments upload successfully, then record the runtime closure fields and close.

[proerror77]

生产故障完整根因记录（2026-08-02 诊断，controller: kimi)

本 issue 最初记录的症状（上传器拒绝 quote_collection_failure kind)已经过时——主机上的 uploader 二进制已更新到 c27c4497（含 #239 以来的 kind 支持）。当前生产故障是三个独立问题叠加，诊断证据全部来自主机只读检查（Cloud Assistant)+ OSS 清单：

1. 市场磁带上传通道卡死（主故障）

• tick-level 配置放大段体积:feat(deploy): tick-level Polymarket tape recording + faster trade backfill #548(quote_sample_ms=0,08-01 部署）使单段从 ~1GB 涨到 4–21GB。
• 磁盘打满:/data(196G）在 08-02 01:49 UTC 达到 100%，上传器报 No space left on device 后崩溃循环（systemd 状态 activating)。当前 spool 积压 141GB / 11 段。
• fix(collector): polymarket-raw-ops upload oss-verify races transient 404 after PUT #571 的 verify 404 竞态放大每次失败:PUT 成功后立即 HEAD 自验 404，每段要重试多轮、每轮全量重传 18GB（约 40 分钟），积压越滚越大。
• 结果：市场磁带（crypto_expiry）自 08-02 01:01 UTC 起无新上传。

2. 参考磁带（reference）覆盖缺口与非 canonical

• polymarket-reference-collector.service 本身在健康运行（每分钟心跳、196 次/分钟成功轮询），但 trade_poll_budget=200 面对 ~2700 个 eligible 市场，deferred_trade_markets 常年 ~2500——交易覆盖先天不足，导致多数段 segment_complete:false → canonical:false(feat(deploy): tick-level Polymarket tape recording + faster trade backfill #539 的 poll budget 200 已部署但不足以清欠账）。
• 2026-08-02 01:26:41–01:59:02 UTC 存在一段 32 分钟的完全无数据洞（恰好覆盖 Acquire a bounded multi-event BTC 5m market cohort in ACK #313 采集窗口），前后段均非 canonical。

3. 上传器二进制缺少两个关键修复

• 无 fix(collector): polymarket-raw-ops upload oss-verify races transient 404 after PUT #571(PUT 后 HEAD 自验无有界重试）——主故障的直接放大器；
• 无 fix(collector): split transport reconnects from manifest request_failures #593（刚合并，manifest 按 integrity/transport 分类）——新磁带在新门禁口径下才算 canonical 的前提。

修复计划（已获 owner 授权）

1. 实现并合并 fix(collector): polymarket-raw-ops upload oss-verify races transient 404 after PUT #571（上传器 oss-verify 有界重试）;
2. 发布新二进制并推送到主机（OSS 中转，主机无 docker);
3. 先验证后清理：逐段确认 OSS 对象存在且哈希一致后清理本已上传的积压，释放磁盘；
4. 换新二进制恢复上传，验证新段 canonical:true、滞留段清零；
5. 记录全部证据后关闭本 issue。参考采集预算不足与 32 分钟洞另行立项跟踪（不影响本 issue 的上传通道合约）。

[proerror77]

ACK controller amendment — exact-main uploader delivery and bounded cutover (2026-08-02)

Controller: /root (this Codex task; owner-authorized by the takeover request in this conversation), sole runtime writer for this production operation.

Authorized target and identity:

• Target: ECS monday-trade-data-26 (i-6we6afeqsvv8uo1ixmyo, ap-northeast-1), service polymarket-market-tape-upload.service, binary /opt/monday/bin/polymarket-raw-ops.
• Source main: 54710b065d6c9395c01fdf9cb0e7863461f433f9 (merged PR fix(collector): retry post-upload OSS verify per artifact on transient 404 #594, including bounded OSS verify retry; PR fix(collector): split transport reconnects from manifest request_failures #593 manifest semantics already in this main).
• Compiler image: crpi-ygobwehhof7qs9m3.ap-northeast-1.personal.cr.aliyuncs.com/wildcard0923/polymarket-evidence-compiler@sha256:1dadd0c2237296dfdbf501c0cb65ac107f251f95b871d9decef6e907b3651433, workflow run 30734745891 succeeded; workflow pulled the digest and verified the embedded revision plus polymarket-raw-ops --version.
• Delivery Job: polymarket-uploader-delivery-533-0802a in ACK cluster c4d1db514b47a4c40995d2ec4bcc8c0ca, namespace monday-research, selector workload=backtest,capacity=spot, no production data mount; it may write only the immutable OSS scratch objects below.
• Delivery output: oss://monday-lob-apne1-1045353359/tmp/binary-delivery/polymarket-raw-ops-54710b065d6c9395c01fdf9cb0e7863461f433f9 and its .sha256 sidecar.

Pre-cutover readback baseline (2026-08-02T05:48:32Z): old binary version c27c4497c5ffb176471dedf27254e39e4c1b4876, old binary SHA-256 1ad9e5d4baa2eb3504a186a92e7ad06969bdd596606e71faa6c0bca720b72a2a; service activating; /data 89% full with 22G available; 13 top-level spool files and 160299869531 bytes under the Polymarket spool; status showed pending_segments=1 (stale while the current attempt was active).

Procedure and fail-closed gates:

1. Run exactly the delivery Job, verify BINARY_DELIVERY_OK, the OSS sidecar, and the downloaded binary SHA before any ECS write.
2. Stop polymarket-market-tape-upload.service; verify no old uploader process remains and preserve all source tapes. Remove no source tape; any stale derived staging cleanup must be separately read back and recorded.
3. Download the immutable delivery object to a host-side temporary path, verify SHA-256 and --version equals 54710b065d6c9395c01fdf9cb0e7863461f433f9, then atomically replace /opt/monday/bin/polymarket-raw-ops and restart the service.
4. Re-read service state, binary identity, disk, spool status, and recent logs after restart; then monitor until retained segments drain. A segment is counted only after its OSS object, manifest, and _SUCCESS readback pass.

Stop rule: stop before ECS replacement on any source/image/Job/OSS identity drift, delivery or SHA mismatch, inability to stop the old process cleanly, or free space falling below 10G; after restart stop and roll back if the new binary is not the exact version, the service fails/re-enters a write loop, a new non-transient failure appears, or disk pressure worsens.

Rollback: stop the service, restore the preserved old binary SHA/version (c27c4497c5ffb176471dedf27254e39e4c1b4876 / 1ad9e5d4baa2eb3504a186a92e7ad06969bdd596606e71faa6c0bca720b72a2a), restart, and retain all source tapes, manifests, status, and delivery artifacts. No production collector/reference configuration, research Jobs, snapshot, evaluator, Paper, Shadow, or Live state is in scope.

[proerror77]

ACK controller amendment — delivery endpoint correction (2026-08-02)

The first delivery Job polymarket-uploader-delivery-533-0802a is retained as immutable evidence. It was scheduled on the authorized workload=backtest,capacity=spot node, but the container never started: kubelet reported pull access denied for the public ACR hostname. No container command ran, no OSS object was written, and no ECS state changed.

Controller: /root remains the sole runtime writer. The amendment below authorizes the one effective delivery attempt; the failed 0802a Job is not retried or mutated.

Target: ECS monday-trade-data-26 (i-6we6afeqsvv8uo1ixmyo), polymarket-market-tape-upload.service, binary /opt/monday/bin/polymarket-raw-ops.
Candidate: same workflow-verified compiler digest and source identity, addressed through the cluster-reachable VPC registry endpoint: crpi-ygobwehhof7qs9m3-vpc.ap-northeast-1.personal.cr.aliyuncs.com/wildcard0923/polymarket-evidence-compiler@sha256:1dadd0c2237296dfdbf501c0cb65ac107f251f95b871d9decef6e907b3651433; source main 54710b065d6c9395c01fdf9cb0e7863461f433f9; workflow run 30734745891.
Controller: /root (this Codex task; owner-authorized by the takeover request).
Stop rule: before any ECS write, stop on image pull failure, version/SHA/delivery readback mismatch, identity drift, or inability to stop the old uploader cleanly. The 0802b Job must emit BINARY_DELIVERY_OK; otherwise no host mutation.
Rollback: no host mutation is made until delivery succeeds; retain 0802a and 0802b evidence. If cutover later fails, restore the preserved c27c4497c5ffb176471dedf27254e39e4c1b4876 binary and its SHA-256 1ad9e5d4baa2eb3504a186a92e7ad06969bdd596606e71faa6c0bca720b72a2a.

Authorized effective delivery Job: polymarket-uploader-delivery-533-0802b in monday-research, selector workload=backtest,capacity=spot, writing only the immutable OSS scratch objects named in the prior amendment.

[proerror77]

Cutover retry note — timer and host credential context (2026-08-02)

The first ECS command (t-jpn6sr3zdtqg934) stopped the uploader service but did not mutate the binary: the root RunCommand context lacked the hft-collector HOME needed by /var/lib/hft-collector/.aliyun/ossutil, so delivery readback failed before download. The unit timer remained active and restarted the old service; a subsequent readback confirmed the target is still SHA-256 1ad9e5d4baa2eb3504a186a92e7ad06969bdd596606e71faa6c0bca720b72a2a, version c27c4497c5ffb176471dedf27254e39e4c1b4876, with no new binary written.

Retry remains within the 0802b authorization. Before any host replacement, /root will stop polymarket-market-tape-upload.timer and the service, verify no uploader process remains, set HOME=/var/lib/hft-collector for the host role profile, and re-read the immutable delivery object and sidecar. The timer is restarted only after the new binary has passed SHA/version checks and the service has been started non-blocking.

Stop rule: abort before replacement on any identity drift, host role readback failure, service/process that cannot be stopped, or free space below 10G. No source spool file is removed.

[proerror77]

Cutover retry note — use the service credential path (2026-08-02)

The second ECS retry stopped neither the binary nor the intended service state because it invoked the raw /var/lib/hft-collector/.aliyun/ossutil ELF. That binary does not read the host Alibaba CLI profile and returned SigningContext.Credentials is null or empty. The production unit invokes /usr/local/bin/aliyun ossutil, which resolves the same role profile through /var/lib/hft-collector/.aliyun/config.json; the delivery Job already passed through that path. No binary or source tape was changed; the service is currently failed/stopped and the timer remains stopped for the next guarded attempt.

The next retry stays within the approved 0802b cutover: stop timer/service, verify old identity, use aliyun ossutil with the service HOME/profile to read the immutable delivery object and sidecar, atomically replace only after SHA/version checks, then start service and timer.

[proerror77]

ACK controller amendment — host-readable immutable delivery prefix (2026-08-02)

The effective delivery Job polymarket-uploader-delivery-533-0802b completed with image/source/version and binary SHA verification, but its tmp/binary-delivery objects are not readable by the ECS role: host aliyun ossutil ls sees the two objects, while GetObject returns 404/NoSuchKey. The same role successfully downloads a lake/raw manifest. No ECS mutation has occurred; the target binary remains the old SHA.

Controller: /root remains the sole runtime writer.

Target: ECS monday-trade-data-26 (i-6we6afeqsvv8uo1ixmyo), polymarket-market-tape-upload.service, /opt/monday/bin/polymarket-raw-ops.
Candidate: source main 54710b065d6c9395c01fdf9cb0e7863461f433f9; compiler image crpi-ygobwehhof7qs9m3-vpc.ap-northeast-1.personal.cr.aliyuncs.com/wildcard0923/polymarket-evidence-compiler@sha256:1dadd0c2237296dfdbf501c0cb65ac107f251f95b871d9decef6e907b3651433; workflow run 30734745891; binary SHA 4da4feb0691b209577f57a7ba28384e5eb720a70d39d0a455330f0f53beb3cda.
Configuration identity: effective delivery Job polymarket-uploader-delivery-533-0802c; selector workload=backtest,capacity=spot; no production mount. It writes only oss://monday-lob-apne1-1045353359/lake/raw/venue=polymarket/dataset=runtime_delivery/date=2026-08-02/hour=05/sha256=4da4feb0691b209577f57a7ba28384e5eb720a70d39d0a455330f0f53beb3cda/polymarket-raw-ops-54710b065d6c9395c01fdf9cb0e7863461f433f9 and its .sha256 sidecar.
Stop rule: do not replace the ECS binary until the host aliyun ossutil cp of both immutable delivery objects succeeds, the downloaded SHA is exactly 4da4feb0691b209577f57a7ba28384e5eb720a70d39d0a455330f0f53beb3cda, and --version is exactly polymarket-raw-ops 54710b065d6c9395c01fdf9cb0e7863461f433f9.
Rollback: 0802a/0802b Jobs and tmp objects remain retained; on any cutover failure restore the preserved old binary SHA 1ad9e5d4baa2eb3504a186a92e7ad06969bdd596606e71faa6c0bca720b72a2a and version c27c4497c5ffb176471dedf27254e39e4c1b4876.
Authorized effective delivery Job: polymarket-uploader-delivery-533-0802c; no other production or research state is in scope.

[proerror77]

Derived staging cleanup amendment — free space before cutover (2026-08-02)

Precondition readback: uploader service failed, timer inactive, no new binary has been installed. The Polymarket spool has 9.6G free. The only stale staging file is /data/monday/spool/polymarket/.upload-staging/session.7891665ac45147d7/market-updates.20260801T193514772831.20260801T19.ndjson, 12624465611 bytes; its complete source /data/monday/spool/polymarket/market-updates.20260801T193514772831.ndjson remains present and is the authoritative retained input.

Controller: /root is the sole runtime writer. Authorized cleanup is limited to that exact derived session directory after a final no-uploader-process check. No top-level source tape, upload-status, OSS object, manifest, or _SUCCESS file may be removed or changed. The cleanup is required to restore the >=10G free-space stop threshold before the new binary starts.
Rollback identity: the source tape and all prior OSS/delivery artifacts remain retained; the removed file is a re-creatable split/staging derivative and is not used as the source of truth.
Stop rule: abort if the exact path, size, source identity, service/timer state, or process state drifts; after removal abort the cutover if free space remains below 10G.