Add compiler scaling fixtures#12
Conversation
|
@coderabbitai[bot] review |
|
Warning Review limit reached
Next review available in: 30 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughChangesCompiler scaling benchmarks
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CIWorkflow
participant BenchCompile
participant FixtureFiles
participant MakoCompiler
participant JSONReport
CIWorkflow->>BenchCompile: invoke benchmark command
BenchCompile->>FixtureFiles: generate fixture sources and metadata
BenchCompile->>MakoCompiler: run cold and warm check/build commands
MakoCompiler-->>BenchCompile: return compile status and elapsed time
BenchCompile->>JSONReport: write benchmark JSON artifacts
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
scripts/bench-compile.py (1)
247-258: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win
source_filesmeans different things in the two JSON schemas.In the per-fixture
fixture.jsonmetadata (Line 252),"source_files"is a list of filenames. In the aggregatemako.compile-bench.v1report (Line 476),"source_files"isfixture.source_files, an int count. Same key name, different type across the two schemas the script emits — confusing for any tooling that consumes both.♻️ Proposed fix
metadata = { "schema": "mako.compile-fixture.v1", "name": name, "shape": shape, "lines": actual, - "source_files": sorted(sources), + "source_file_names": sorted(sources), }Also applies to: 472-478
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/bench-compile.py` around lines 247 - 258, Rename the aggregate report’s source-file count field in the reporting logic around the aggregate metadata construction (near the use of fixture.source_files) to distinguish it from the per-fixture source_files filename list. Keep the per-fixture metadata key unchanged, and update the aggregate schema/value consistently so it uses an explicit count-oriented key while preserving the integer count.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/bench-compile.py`:
- Around line 307-322: Add a finite timeout to the subprocess.run calls in
run_once and command_output, using the benchmark’s existing timeout
configuration or a shared constant where available. Preserve current success
handling and ensure timeout failures propagate clearly instead of allowing a
hung compiler to block indefinitely.
---
Nitpick comments:
In `@scripts/bench-compile.py`:
- Around line 247-258: Rename the aggregate report’s source-file count field in
the reporting logic around the aggregate metadata construction (near the use of
fixture.source_files) to distinguish it from the per-fixture source_files
filename list. Keep the per-fixture metadata key unchanged, and update the
aggregate schema/value consistently so it uses an explicit count-oriented key
while preserving the integer count.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bd6818c7-eff5-4148-96a6-9715bbd6d1c9
📒 Files selected for processing (6)
.github/workflows/ci.ymlCHANGELOG.mdbenchmarks/compile/README.mdbenchmarks/compile/fixtures.jsondocs/PERFORMANCE.mdscripts/bench-compile.py
7d71122 to
3c02bfd
Compare
|
@coderabbitai[bot] review |
|
✅ Action performedReview finished.
|
2 participants
Adds deterministic compiler-scaling fixtures at 1k, 10k, and 100k lines across single-file, multi-file, generic-heavy, and backend-shaped projects.
The runner records cold and cached checks, supports optional full builds, and writes attributable JSON reports. CI validates the full generated matrix on Linux, exercises 10k single and generic builds, and uploads a 1k smoke report on every native platform without enforcing timing thresholds.
Verified with
cargo test --bin mako(78 passed), the full 12-fixture check matrix, and the 10k full-build smoke.Summary by CodeRabbit
New Features
Documentation
Chores