fix(collector): raise retained trade ID cap to 4M and make it configurable

[proerror77]

Change contract

Raise the reference collector's retained trade-ID dedupe cap 1M → 4M (default), add --max-retained-trade-ids for tuning. No unit/cmdline changes, so gate identity pins are untouched.

Production evidence

2026-08-01 shadow gate invocation e50c2f50 (candidate e5c661ed): the collector bailed with collector state exceeds configured market or retained trade ID limit after ~15 minutes — at the current catalog, a budget-200 backfill accumulates ~42k IDs/min (953,566 recorded in the state checkpoint), and every restart re-hits the cap within minutes, so the shadow can never complete a gate observation window.

Memory math (in the constant's comment)

~150-200B per ID in memory → 4M IDs cost well under the units' 1536M/2048M watermarks (#574).

Out of scope

• Pruning semantics for incomplete markets (a real design question, separate issue if desired)

Dependency / merge order

None. Required before the next gate attempt — the current shadow crash-loops on this cap.

Focused validation

• cargo test -p hft-collector --locked: 262 lib + 72 bin tests green

Rollout / rollback impact

Config default only. Rollback = revert.

Issue relationship

Refs #553

Summary by CodeRabbit

• New Features

  • Added a configurable limit for retained trade IDs in the reference data collector.
  • Increased the default retained trade-ID capacity from 1 million to 4 million.
  • Added a command-line option to customize the retention limit.

• Bug Fixes

  • Improved validation to consistently honor the configured trade-ID retention capacity during collection.

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 47 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bd6481a7-cf35-46e5-bd1e-f95e0b5f035f

📥 Commits

Reviewing files that changed from the base of the PR and between fd91aa5 and e115dc4.

📒 Files selected for processing (2)

• rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs
• rust_hft/tools/collector/src/polymarket_raw.rs

📝 Walkthrough

Walkthrough

The collector increases the default retained trade-ID capacity to 4,000,000. ReferenceConfig exposes this limit, and the collect-reference CLI can override it. Startup and per-cycle validation use the configured value.

Changes

Retained trade-ID capacity

Layer / File(s)	Summary
Collector capacity configuration rust_hft/tools/collector/src/polymarket_raw.rs	The collector defines a public 4,000,000 default, adds ReferenceConfig.max_retained_trade_ids, and uses the configured limit for startup and per-cycle validation.
CLI capacity wiring rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs	The collect-reference command accepts max_retained_trade_ids and passes it to ReferenceConfig.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• proerror77/monday#34: Introduces the ReferenceConfig and collect-reference CLI extended by this change.
• proerror77/monday#37: Modifies the same collector configuration and CLI wiring.
• proerror77/monday#86: Modifies ReferenceConfig and retained trade-ID handling in polymarket_raw.rs.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the cap increase and the addition of configurability, which are the main changes.
Description check	✅ Passed	The description covers the change, rationale, issue relationship, scope, validation, dependencies, and rollback, but omits the Scope exception heading.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/retained-ids-flag

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs`:
- Around line 61-62: Update the focused test
collect_reference_cli_uses_the_library_discovery_capacity_default to assert that
the CLI’s max_retained_trade_ids value matches DEFAULT_MAX_RETAINED_TRADE_IDS,
covering the new option and detecting future CLI/library default mismatches.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0562421c-1852-42dc-9174-a3c402b671c0

📥 Commits

Reviewing files that changed from the base of the PR and between f427c34 and fd91aa5.

📒 Files selected for processing (2)

• rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs
• rust_hft/tools/collector/src/polymarket_raw.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd91aa518b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".