Skip to content

fix(pm-tape): recorder restart-loops at tick level — CPU/memory quota + lag fail-closed wrong for pure recorder #581

Description

@proerror77

Summary

After switching polymarket-market-tape to tick-level recording (record_market_updates_quote_sample_ms = 0, kinds + spot_price/agg_trade/l2), the recorder restart-loops: 13 restarts in ~25 minutes, each restart losing 5–60s of tape.

Root cause (verified on host i-6we6afeqsvv8uo1ixmyo, 2026-08-02)

  1. CPU starvation: cgroup cpu.stat shows nr_throttled 19 / nr_periods 38 (50% of periods throttled) under CPUQuota=30%. Memory peaks sit exactly at MemoryHigh=384M, triggering direct-reclaim stalls. The consumer falls behind, the 8192-message broadcast ring overflows.
  2. Wrong failure semantic for a pure recorder: LiveFeed treats any Lagged as fatal (fail-closed for trading). For the tape recorder (strategy_variant = "noop", DryRun), one burst → LiveFeed lagged; closing feed fail-closed → runtime exits → systemd restarts → bigger data gap than the lag itself.

Evidence

WARN ploy_strategy_bundles::feed::live: LiveFeed lagged; closing feed fail-closed skipped=49
systemd[1]: polymarket-market-tape.service: Scheduled restart job, restart counter is at 10.

Change contract

Recorder survives tick-level load without restart-looping; trading deployments keep fail-closed lag semantics.

  • feed_broadcast_capacity runtime config key (default 8192 unchanged).
  • feed_lag_policy = "fail_closed" | "skip_and_continue" runtime config key (default fail_closed unchanged); tape config opts into skip_and_continue with warn-logged gap counts.
  • Unit quotas for the tape service raised to CPUQuota=100%, MemoryHigh=768M, MemoryMax=1G (host: 7.4GB RAM, 5.1GB available).
  • Contract test pins the new tape config keys and quota headroom.

Out of scope

Acceptance

  • cargo test -p ploy-strategy-bundles -p ploy-strategy-runtime passes with new lag-policy tests.
  • Contract test pins feed_lag_policy, feed_broadcast_capacity, CPUQuota=100%, MemoryMax=1G.
  • Deployed recorder runs ≥2h at tick level with zero restarts.

Runtime control

  • Target: polymarket-market-tape.service on Tokyo ECS i-6we6afeqsvv8uo1ixmyo (/opt/monday/releases/polymarket-market-recorder/*, config /etc/monday/polymarket-market-tape.toml).
  • Candidate: PR fix(pm-tape): survive tick-level load without restart-looping #582 (codex/pm-tape-lag): feed_broadcast_capacity = 65536, feed_lag_policy = "skip_and_continue", unit CPUQuota=100% / MemoryHigh=768M / MemoryMax=1G; recorder release built from the merged SHA.
  • Controller: kimi main session (authorized by proerror77); no other agent controls this lane (Codex agent confirmed closed).
  • Stop rule: stop mutation if the recorder restart-loops again (NRestarts > 0 within 10 min), MemoryCurrent stays above 900M, tape gaps exceed 60s outside rotation, or failed_segments becomes non-empty.
  • Rollback: remove /etc/systemd/system/polymarket-market-tape.service.d/10-resources.conf, restore previous unit quotas (CPUQuota=30%, MemoryHigh=384M, MemoryMax=512M) and previous toml (quote_sample_ms = 1000, reduced kinds) from the .bak-* backup, daemon-reload + restart; recorder release rollback pins the previous release SHA.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingready-for-humanHuman implementation or judgment is requiredruntimeRequires deployment, runtime mutation, or live evidence for closure

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions