Skip to content

ci(rust): parallelize recorder contract and protect main cache production - #569

Merged
proerror77 merged 6 commits into
mainfrom
codex/ci-finishing-items
Aug 1, 2026
Merged

ci(rust): parallelize recorder contract and protect main cache production#569
proerror77 merged 6 commits into
mainfrom
codex/ci-finishing-items

Conversation

@proerror77

@proerror77 proerror77 commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Change contract

Two finishing items for the CI layering (per #558's final review):

  1. ci(rust): make market-recorder release contract a parallel heavy job with docker layer caching #568: the market-recorder release contract (~5.5min docker build, structurally unable to use host sccache) moves from a serial step inside Rust Workspace into its own parallel job market_recorder_contract, gated on control == 'true' and a member of ci-gate. Rust Workspace critical path drops by ~5.5min on control-lane runs.
  2. Cache-production stability: concurrency.cancel-in-progress is now false on main (still true on PR branches) so main runs always complete and keep feeding rust-cache/sccache to everyone.

Out of scope

Dependency / merge order

None; stacks on #566 (merged). Scope-contract pins updated and PASS.

Focused validation

  • .github/scripts/test-select-rust-ci-scope.sh PASS with new pins (recorder job block, gate membership, absence from rust/fast-gates)
  • ci.yml YAML parse OK
  • Evidence it addresses: Rust Workspace 11m20s with serial recorder (review measurement); expected ~5.5min drop on control runs

Rollout / rollback impact

CI-only. Rollback = revert.

Issue relationship

Closes #568

Summary by CodeRabbit

  • Bug Fixes

    • Improved continuous integration reliability by ensuring market recorder release checks run in their dedicated validation stage.
    • Added coverage for raw operations control-plane checks in the primary Rust validation suite.
    • Improved CI behavior when build-cache services are unavailable.
  • Chores

    • Updated CI run handling so changes on the main branch are preserved while superseded non-main runs may be canceled.
    • Included the market recorder validation stage in the overall CI completion gate.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 39 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c8609589-1c5e-43b7-9e8c-c08450488fcb

📥 Commits

Reviewing files that changed from the base of the PR and between a8a3c8c and 0453382.

📒 Files selected for processing (2)
  • ccpm
  • rust_hft/serena
📝 Walkthrough

Walkthrough

The workflow preserves main runs, tolerates unavailable sccache, moves the market recorder release contract to a dedicated job, adds that job to ci-gate, and updates Rust CI scope tests.

Changes

Rust CI topology

Layer / File(s) Summary
CI concurrency and cache resilience
.github/workflows/ci.yml
Concurrency cancellation remains enabled for non-main branches but is disabled for main. Rust and fast-lane cache dimension checks allow an unavailable sccache.
Dedicated market recorder CI job
.github/workflows/ci.yml
The market recorder release contract moves out of the main Rust job into a dedicated 20-minute job. ci-gate depends on the new job.
CI scope contract validation
.github/scripts/test-select-rust-ci-scope.sh
The script verifies raw-ops coverage, dedicated market recorder job coverage, CI gate membership, and exclusion from the main Rust job.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CIWorkflow
  participant RustJob
  participant MarketRecorderContract
  participant CIGate
  CIWorkflow->>RustJob: Run Rust checks without market recorder release
  CIWorkflow->>MarketRecorderContract: Run market recorder release contract
  RustJob-->>CIGate: Report Rust job result
  MarketRecorderContract-->>CIGate: Report contract job result
  CIGate-->>CIWorkflow: Evaluate required job results
Loading

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The concurrency and sccache-detection changes are not covered by linked issue #568 and extend beyond the recorder-contract objective. Link the cache-production changes to a supporting issue or remove them from this PR.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the recorder contract parallelization and main cache-production changes.
Description check ✅ Passed The description covers the change, issue, scope, dependencies, validation, rollout, and rollback; only the scope-exception section is omitted.
Linked Issues check ✅ Passed The PR moves the recorder contract out of Rust Workspace, creates a parallel job, and preserves ci-gate membership as required by #568.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/ci-finishing-items

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread .github/workflows/ci.yml Fixed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d06157281

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/ci.yml
@proerror77
proerror77 force-pushed the codex/ci-finishing-items branch from 7d06157 to 4691f9b Compare August 1, 2026 07:36

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
.github/scripts/test-select-rust-ci-scope.sh (1)

220-225: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Assert the control gate in the scope contract.

The current checks verify the job name, release script, and Rust-job exclusion. They pass if market_recorder_contract runs on every scope. Add assertions for needs: scope and if: needs.scope.outputs.control == 'true' to enforce the control-lane requirement in .github/workflows/ci.yml Line 444.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/scripts/test-select-rust-ci-scope.sh around lines 220 - 225, Extend
the market_recorder_contract assertions around recorder_block to require the job
definition’s needs: scope dependency and the conditional gate if:
needs.scope.outputs.control == 'true'. Keep the existing job-name,
release-script, and Rust-job exclusion checks unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/test-select-rust-ci-scope.sh:
- Line 178: Update the dependency assertion in test-select-rust-ci-scope.sh to
run after job_block() extracts the ci-gate block, and grep that extracted block
rather than the full workflow. Preserve the existing market_recorder_contract
dependency check while ensuring it specifically validates the ci-gate job.

In @.github/workflows/ci.yml:
- Around line 448-449: Update the Checkout step in the release/Docker build
workflow to set persist-credentials to false, preventing the GitHub token from
remaining in .git/config before privileged Docker builds; preserve fetch-depth:
0 only in checkout steps that require full history.

---

Nitpick comments:
In @.github/scripts/test-select-rust-ci-scope.sh:
- Around line 220-225: Extend the market_recorder_contract assertions around
recorder_block to require the job definition’s needs: scope dependency and the
conditional gate if: needs.scope.outputs.control == 'true'. Keep the existing
job-name, release-script, and Rust-job exclusion checks unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 02a26e16-c15d-4c45-b770-714f616898ad

📥 Commits

Reviewing files that changed from the base of the PR and between 5dea0da and 4691f9b.

📒 Files selected for processing (2)
  • .github/scripts/test-select-rust-ci-scope.sh
  • .github/workflows/ci.yml

Comment thread .github/scripts/test-select-rust-ci-scope.sh Outdated
Comment thread .github/workflows/ci.yml
proerror77 pushed a commit that referenced this pull request Aug 1, 2026
…ng sccache binary

- market_recorder_contract checkout now uses persist-credentials: false
  so the GITHUB_TOKEN cannot leak into the docker build context
  (zizmor artipacked finding on #569).
- The gate-membership pin is scoped to the ci-gate needs block.
- Record cache dimensions tolerates a missing sccache binary
  (continue-on-error fallback path).

Refs #568
Comment thread .github/workflows/ci.yml Fixed
Comment thread .github/workflows/ci.yml
proerror77 pushed a commit that referenced this pull request Aug 1, 2026
The ccpm entry is a gitlink with no .gitmodules registration (added
accidentally in 33ecc78). It breaks actions/checkout whenever a job
runs the submodule sshCommand cleanup path (observed failing the
market_recorder_contract job on #569). Nothing references it.
Sonic Shih added 4 commits August 1, 2026 16:41
…tion

- The market-recorder release contract (a ~5.5min docker build that
  cannot use host sccache) moves out of the serial Rust Workspace job
  into its own parallel job, keeping ci-gate membership (#568).
- concurrency.cancel-in-progress is now false on main: PR runs still
  supersede, but main runs finish so rust-cache/sccache keep
  producing for all branches.

Refs #568
…ng sccache binary

- market_recorder_contract checkout now uses persist-credentials: false
  so the GITHUB_TOKEN cannot leak into the docker build context
  (zizmor artipacked finding on #569).
- The gate-membership pin is scoped to the ci-gate needs block.
- Record cache dimensions tolerates a missing sccache binary
  (continue-on-error fallback path).

Refs #568
The default checkout died on the repo's gitlink-without-.gitmodules
ccpm entry during submodule cleanup; every other job in this workflow
uses fetch-depth: 0 and passes.

Refs #568
@proerror77
proerror77 force-pushed the codex/ci-finishing-items branch from 857f825 to a8a3c8c Compare August 1, 2026 08:41
The ccpm entry is a gitlink with no .gitmodules registration (added
accidentally in 33ecc78). It breaks actions/checkout whenever a job
runs the submodule sshCommand cleanup path (observed failing the
market_recorder_contract job on #569). Nothing references it.
@proerror77
proerror77 enabled auto-merge (squash) August 1, 2026 08:53
Second gitlink without .gitmodules registration, surfaced after the
ccpm removal unblocked the first. Also unreferenced.

Refs #568
@proerror77
proerror77 merged commit ff8ad55 into main Aug 1, 2026
44 checks passed
@proerror77
proerror77 deleted the codex/ci-finishing-items branch August 1, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(rust): make market-recorder release contract a parallel heavy job with docker layer caching

2 participants