collector: gate CEX L2 continuity evidence - #265
Conversation
📝 WalkthroughWalkthroughThe gate now performs strict LOB continuity verification, records continuity metrics in round-trip evidence, validates them for spot and USDM markets, and emits schema v3 payloads with expanded test coverage. ChangesLOB continuity gate
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GateScript
participant Archiver
participant OSSManifest
participant GatePolicy
GateScript->>Archiver: request strict segment verification
Archiver->>OSSManifest: verify trade and LOB continuity
OSSManifest-->>GateScript: return continuity metrics
GateScript->>GatePolicy: submit schema v3 gate evidence
GatePolicy-->>GateScript: accept or reject evidence
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10f8711833
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| and .lob_continuity.contract == "binance.lob_continuity.v1" | ||
| and .lob_continuity.capture_session_id == $session_id |
There was a problem hiding this comment.
For every manifest whose end time reaches the gate window, this second jq invocation references $session_id without defining it. The --arg session_id on the earlier discovery invocation is local to that separate process, so jq exits with '$session_id is not defined' and the production shadow gate always takes the failure path before verifying any OSS segments. Add the argument to this invocation or combine the checks.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2bf102f: the session argument is now bound on the same jq invocation that evaluates the LOB continuity contract, with a focused regression assertion.
| --argjson gap_from_previous_ns "$gap_ns" \ | ||
| --argjson bytes "$bytes" \ | ||
| --argjson agg_trade_count "$agg_trade_count" \ | ||
| --argjson lob_continuity "$lob_continuity" \ |
There was a problem hiding this comment.
With the configured full-catalog run, lob_continuity.symbols contains hundreds to more than 1,000 verbose per-symbol objects, so the compact summary is substantially larger than Linux's per-argument limit (a representative 1,000-symbol summary is about 340 KB). Expanding the whole value into this single --argjson argument therefore makes the shell fail to execute jq with Argument list too long, preventing round-trip evidence from being produced even after the earlier jq-variable issue is fixed. Have jq read the manifest/summary from a file instead of passing it through argv.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2bf102f: the full continuity summary is now read directly from the canonical manifest via --slurpfile; the control-plane test rejects any --argjson lob_continuity regression.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
deployment/aliyun/test-rust-lob-control-plane.sh (1)
1-1: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate schema-v3 LOB-evidence fixture construction across two test scripts. Both scripts independently hand-build the same
market/usdm_market/gate JSON shape (session id, reconnect-boundary pattern, latency/level bounds, usdm override map) with only literal values differing. One shared root cause: no common fixture helper for schema-v3 LOB continuity evidence.
deployment/aliyun/test-rust-lob-control-plane.sh#L80-113: extract the market/usdm_market/gate JSON construction into a shared shell helper (e.g. a sourcedlob-gate-fixture.sh) parameterized by session id, symbol counts, and hashes.deployment/aliyun/test-rust-lob-release-adoption.sh#L146-185: source and call the same shared helper instead of re-implementing the fixture shape here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@deployment/aliyun/test-rust-lob-control-plane.sh` at line 1, Extract the duplicated schema-v3 LOB evidence JSON construction from the test flow in test-rust-lob-control-plane.sh into a shared sourced shell helper, parameterized by session ID, symbol counts, and hashes. Update test-rust-lob-release-adoption.sh to source and call that helper, preserving each script’s existing literal values while removing its local market/usdm_market/gate fixture construction.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@deployment/aliyun/test-rust-lob-control-plane.sh`:
- Line 1: Extract the duplicated schema-v3 LOB evidence JSON construction from
the test flow in test-rust-lob-control-plane.sh into a shared sourced shell
helper, parameterized by session ID, symbol counts, and hashes. Update
test-rust-lob-release-adoption.sh to source and call that helper, preserving
each script’s existing literal values while removing its local
market/usdm_market/gate fixture construction.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: cf398508-517c-49fb-b6c1-9239ffaa61b5
📒 Files selected for processing (5)
deployment/aliyun/host-rust-lob-shadow-gate.shdeployment/aliyun/rust-lob-shadow-gate-policy.jqdeployment/aliyun/test-rust-lob-control-plane.shdeployment/aliyun/test-rust-lob-release-adoption.shrust_hft/tools/collector/src/bin/binance-lob-archiver.rs
CI classification before merge:
This is classified as an unrelated runner/test instability, not a #224 behavior failure. No trading-host code or test change is being mixed into this collector PR. The production shadow gate remains unrun and issue #224 stays open. |
1 participant
Change contract
Require the packaged Binance collector verifier and Rust LOB shadow gate to prove sealed
binance.lob_continuity.v1continuity for every canonical OSS round-trip before a candidate can be promoted.Out of scope
Dependencies and merge order
Depends on merged PR #264; merge this PR after #264.
Focused validation
cargo test -p hft-collector --bin binance-lob-archiver(40 passed)bash deployment/aliyun/test-rust-lob-control-plane.shbash deployment/aliyun/test-rust-lob-release-adoption.shshellcheck deployment/aliyun/host-rust-lob-shadow-gate.sh deployment/aliyun/test-rust-lob-control-plane.sh deployment/aliyun/test-rust-lob-release-adoption.shbash -nfor the three changed shell scriptsgit diff --checkThe repository-wide
cargo fmt --all -- --checkstill reports pre-existing formatting drift outside this change; the new Rust hunk was aligned to rustfmt output.Rollout and rollback
No production mutation in this PR. A later named single controller must package this merged gate, run the isolated one-hour shadow gate, and read back canonical artifacts before cutover. Roll back this layer by reverting this PR; v2 evidence then remains acceptable only to the prior policy.
Scope exception
None.
Closes no issue: #224 remains open until the live shadow gate and canonical artifact readback are completed.
Summary by CodeRabbit
New Features
Bug Fixes
Tests