You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After switching polymarket-market-tape to tick-level recording (record_market_updates_quote_sample_ms = 0, kinds + spot_price/agg_trade/l2), the recorder restart-loops: 13 restarts in ~25 minutes, each restart losing 5–60s of tape.
Root cause (verified on host i-6we6afeqsvv8uo1ixmyo, 2026-08-02)
CPU starvation: cgroup cpu.stat shows nr_throttled 19 / nr_periods 38 (50% of periods throttled) under CPUQuota=30%. Memory peaks sit exactly at MemoryHigh=384M, triggering direct-reclaim stalls. The consumer falls behind, the 8192-message broadcast ring overflows.
Wrong failure semantic for a pure recorder: LiveFeed treats any Lagged as fatal (fail-closed for trading). For the tape recorder (strategy_variant = "noop", DryRun), one burst → LiveFeed lagged; closing feed fail-closed → runtime exits → systemd restarts → bigger data gap than the lag itself.
Evidence
WARN ploy_strategy_bundles::feed::live: LiveFeed lagged; closing feed fail-closed skipped=49
systemd[1]: polymarket-market-tape.service: Scheduled restart job, restart counter is at 10.
Change contract
Recorder survives tick-level load without restart-looping; trading deployments keep fail-closed lag semantics.
cargo test -p ploy-strategy-bundles -p ploy-strategy-runtime passes with new lag-policy tests.
Contract test pins feed_lag_policy, feed_broadcast_capacity, CPUQuota=100%, MemoryMax=1G.
Deployed recorder runs ≥2h at tick level with zero restarts.
Runtime control
Target: polymarket-market-tape.service on Tokyo ECS i-6we6afeqsvv8uo1ixmyo (/opt/monday/releases/polymarket-market-recorder/*, config /etc/monday/polymarket-market-tape.toml).
Candidate: PR fix(pm-tape): survive tick-level load without restart-looping #582 (codex/pm-tape-lag): feed_broadcast_capacity = 65536, feed_lag_policy = "skip_and_continue", unit CPUQuota=100% / MemoryHigh=768M / MemoryMax=1G; recorder release built from the merged SHA.
Controller: kimi main session (authorized by proerror77); no other agent controls this lane (Codex agent confirmed closed).
Stop rule: stop mutation if the recorder restart-loops again (NRestarts > 0 within 10 min), MemoryCurrent stays above 900M, tape gaps exceed 60s outside rotation, or failed_segments becomes non-empty.
Rollback: remove /etc/systemd/system/polymarket-market-tape.service.d/10-resources.conf, restore previous unit quotas (CPUQuota=30%, MemoryHigh=384M, MemoryMax=512M) and previous toml (quote_sample_ms = 1000, reduced kinds) from the .bak-* backup, daemon-reload + restart; recorder release rollback pins the previous release SHA.
Summary
After switching
polymarket-market-tapeto tick-level recording (record_market_updates_quote_sample_ms = 0, kinds +spot_price/agg_trade/l2), the recorder restart-loops: 13 restarts in ~25 minutes, each restart losing 5–60s of tape.Root cause (verified on host i-6we6afeqsvv8uo1ixmyo, 2026-08-02)
cpu.statshowsnr_throttled 19 / nr_periods 38(50% of periods throttled) underCPUQuota=30%. Memory peaks sit exactly atMemoryHigh=384M, triggering direct-reclaim stalls. The consumer falls behind, the 8192-message broadcast ring overflows.LiveFeedtreats anyLaggedas fatal (fail-closed for trading). For the tape recorder (strategy_variant = "noop", DryRun), one burst →LiveFeed lagged; closing feed fail-closed→ runtime exits → systemd restarts → bigger data gap than the lag itself.Evidence
Change contract
Recorder survives tick-level load without restart-looping; trading deployments keep fail-closed lag semantics.
feed_broadcast_capacityruntime config key (default 8192 unchanged).feed_lag_policy = "fail_closed" | "skip_and_continue"runtime config key (defaultfail_closedunchanged); tape config opts intoskip_and_continuewith warn-logged gap counts.CPUQuota=100%,MemoryHigh=768M,MemoryMax=1G(host: 7.4GB RAM, 5.1GB available).Out of scope
l2-kind upload validation, backlog drain (fix(collector): orphan segment recovery stalls startup unboundedly (usdm 3-day outage root cause) #552 orphans, feat(research): accept tick-level + mixed-kind Polymarket tapes in research import and evidence validation #542 research-lane acceptance).Acceptance
cargo test -p ploy-strategy-bundles -p ploy-strategy-runtimepasses with new lag-policy tests.feed_lag_policy,feed_broadcast_capacity,CPUQuota=100%,MemoryMax=1G.Runtime control
polymarket-market-tape.serviceon Tokyo ECSi-6we6afeqsvv8uo1ixmyo(/opt/monday/releases/polymarket-market-recorder/*, config/etc/monday/polymarket-market-tape.toml).codex/pm-tape-lag):feed_broadcast_capacity = 65536,feed_lag_policy = "skip_and_continue", unitCPUQuota=100%/MemoryHigh=768M/MemoryMax=1G; recorder release built from the merged SHA.NRestarts > 0within 10 min),MemoryCurrentstays above 900M, tape gaps exceed 60s outside rotation, orfailed_segmentsbecomes non-empty./etc/systemd/system/polymarket-market-tape.service.d/10-resources.conf, restore previous unit quotas (CPUQuota=30%,MemoryHigh=384M,MemoryMax=512M) and previous toml (quote_sample_ms = 1000, reduced kinds) from the.bak-*backup,daemon-reload+ restart; recorder release rollback pins the previous release SHA.