feat(bybit): governed options archiver with disk gate + bounded backoff - #737
Conversation
Bring back the Bybit v5 options market-data collector as a repo-governed collector bin after the 2026-08-05/06 Aliyun disk-full incident. Fix the three defects that caused unbounded local growth: 1. Fail-closed disk/spool gates: MIN_FREE_GB (default 20.0) and BYBIT_OPTIONS_SPOOL_MAX_BYTES (default 50 GiB) bail the writer before segment open/rotate and the uploader before compress; disk_free_gb, disk_warning, spool_warning surface in health.json. 2. Uploader recycles the source .ndjson only after verified OSS readback via an .uploaded.json marker (idempotent), keeps .zst as a bounded local fallback, and sweeps it after BYBIT_OPTIONS_LOCAL_ZST_RETENTION_SECONDS. 3. WS handshake carries a monday-bybit-options-archiver/<rev> User-Agent plus Origin and app_id, and reconnect uses bounded exponential backoff (backoff*2).min(30) reset to 1s on success. Add the governed deployment lane mirroring the binance lane: staging deploy, one-hour shadow gate, cutover, systemd units with RuntimeMaxSec and the fail-closed env, runtime health policy, shadow-gate policy, control-plane freshness lib, host scripts, and test harnesses.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 5 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (14)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…740) * ci(acr-publish): carry bybit-options-archiver in the collector image #737 added the governed bybit-options-archiver bin but the canonical ACR pipeline never learned about it: the image Dockerfile built and installed only the five pre-existing collector binaries, and the publish workflow's bare-metal artifact extraction skipped the bybit binary entirely. Without this the governed Bybit Options lane has no canonical build path. - Dockerfile.binance-lob-archiver: build --bin bybit-options-archiver with the same --locked --no-default-features flags, install to /out/bin, and copy into the runtime image - acr-publish.yml: extract artifact/bybit-options-archiver, verify its --version binds the source revision, and record its sha256 alongside the other binaries - test-bybit-options-release-contract.sh: assert the image and workflow carry the bin, mirroring the usdm-reference contract test * test(bybit-options): silence SC2016 on the literal workflow expression --------- Co-authored-by: Sonic Shih <sonic.shih@mandonothing.com>
1 participant
Change contract
Bring the
bybit-options-archiverback as a repo-governed collector bin andfix the three defects that caused the 2026-08-05/06 disk-full incident. The
source is rewritten from the unmerged
codex/ecs-venue-collectors-20260805branch (773 lines) into a 1255-line governed bin: (1) fail-closed disk gate
(
MIN_FREE_GBdefault 20.0,BYBIT_OPTIONS_SPOOL_MAX_BYTESdefault 50 GiB,enforced before first open, rotation, and upload_pending); (2) uploader recycles
the raw
.ndjsononly after a readback-verified.uploaded.jsonmarker(idempotent recovery), keeping
.zstas a bounded fallback swept byBYBIT_OPTIONS_LOCAL_ZST_RETENTION_SECONDS; (3) bounded WS reconnect(
(backoff*2).min(30)reset on success) withUser-Agent/Origin/app_idheaders. Adds a full governance lane (shadow-gate, cutover, runtime-health +
shadow-gate policies, control-plane lib, staging-only deploy, two test
harnesses) mirroring the binance lane.
Issue relationship
None
Out of scope
disabled until a separately reviewed cutover; the monitoring contract asserts
it remains disabled until then).
Dependencies and merge order
None. Based on
a74ee6f4; README region (~line 435) is disjoint from the otherbranches' README regions, so merges are clean in any order.
Focused validation
cargo test -p hft-collector --bin bybit-options-archiver --locked— 12/12 PASScargo clippy -p hft-collector --bin bybit-options-archiver --all-targets --no-deps --locked -- -D warnings— cleantest-bybit-options-release-contract.sh,test-bybit-options-shadow-gate.sh— PASSbash -non all 6 bybit shell scripts;git diff --check— cleansystemd-run/systemd unit behavior, realdf -Pk, and OSS readback require host verification (not run here).Rollout and rollback
Production impact is
Noneuntil the bybit lane is shadow-gated and cut over onthe host by a named controller (per README); the archiver unit remains disabled
otherwise. Rollback: revert the commit; the unit never becomes live without the
governed cutover.
Scope exception
14 files / 3168 non-generated lines exceeds the 750-line default. This is one
inseparable safety contract: a single collector binary whose three defect fixes
are one rollback unit, plus the governance lane that makes its live cutover
fail-closed. Named reviewer (repository owner) approved the full diff in
docs/reviews/2026-08-07-incident-remediation-diff-review.md.