Use range-based benchmark comparison in PR comments#144
Merged
Conversation
Contributor
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 1 minutes and 6 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughThis PR replaces single-value benchmark comparison in CI with a range-aware approach. Benchmarks now include minimum and maximum ops/sec values alongside point estimates, and the comparison logic classifies changes as improved/regressed based on range overlap rather than fixed percentage thresholds, with classification markers displayed in the delta column. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Suite Timing
Measured on ubuntu-latest x64. |
Contributor
Benchmark Results263 benchmarks Interpreted: 🟢 42 improved · 🔴 74 regressed · 147 unchanged · avg -0.5% arraybuffer.js — Interp: 🟢 3, 🔴 3, 8 unch. · avg +0.0% · Bytecode: 🔴 14 · avg -8.4%
arrays.js — Interp: 🟢 9, 🔴 4, 6 unch. · avg +0.8% · Bytecode: 🔴 19 · avg -10.5%
async-await.js — Interp: 🟢 1, 🔴 1, 4 unch. · avg +0.1% · Bytecode: 🔴 6 · avg -9.3%
classes.js — Interp: 🟢 5, 🔴 5, 21 unch. · avg +0.3% · Bytecode: 🟢 1, 🔴 18, 12 unch. · avg -6.9%
closures.js — Interp: 🟢 5, 🔴 1, 5 unch. · avg +0.4% · Bytecode: 🔴 7, 4 unch. · avg -4.6%
collections.js — Interp: 🟢 3, 🔴 2, 7 unch. · avg +0.7% · Bytecode: 🟢 1, 🔴 10, 1 unch. · avg -4.8%
destructuring.js — Interp: 🔴 15, 7 unch. · avg -3.0% · Bytecode: 🔴 19, 3 unch. · avg -7.7%
fibonacci.js — Interp: 🟢 1, 7 unch. · avg +0.4% · Bytecode: 🔴 8 · avg -12.7%
for-of.js — Interp: 🔴 1, 6 unch. · avg -0.9% · Bytecode: 🔴 7 · avg -6.0%
helpers/bench-module.js — Interp: 0 · Bytecode: 0
iterators.js — Interp: 🟢 7, 13 unch. · avg +1.3% · Bytecode: 🔴 20 · avg -12.6%
json.js — Interp: 🟢 3, 🔴 2, 15 unch. · avg +0.0% · Bytecode: 🟢 1, 🔴 19 · avg -7.4%
jsx.jsx — Interp: 🔴 6, 15 unch. · avg -0.9% · Bytecode: 🔴 18, 3 unch. · avg -3.6%
modules.js — Interp: 🔴 5, 4 unch. · avg -1.9% · Bytecode: 🔴 8, 1 unch. · avg -8.2%
numbers.js — Interp: 🔴 6, 5 unch. · avg -2.6% · Bytecode: 🔴 11 · avg -10.2%
objects.js — Interp: 🔴 4, 3 unch. · avg -2.8% · Bytecode: 🔴 6, 1 unch. · avg -3.2%
promises.js — Interp: 🔴 3, 9 unch. · avg -0.4% · Bytecode: 🔴 12 · avg -4.9%
strings.js — Interp: 🟢 1, 🔴 5, 5 unch. · avg -0.5% · Bytecode: 🟢 1, 🔴 9, 1 unch. · avg -2.6%
typed-arrays.js — Interp: 🟢 4, 🔴 11, 7 unch. · avg -1.4% · Bytecode: 🟢 1, 🔴 21 · avg -8.7%
Measured on ubuntu-latest x64. Benchmark ranges compare cached main-branch min/max ops/sec with the PR run; overlapping ranges are treated as unchanged noise. Percentage deltas are secondary context. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/pr.yml:
- Around line 396-405: calculateChange can return null when baseEntry.opsPerSec
is zero which later leads fmtPercent(fmtDelta(...)) to call null.toFixed and
throw; fix by making fmtDelta (and/or fmtPercent) robust to null change values:
update fmtDelta to check if the change argument is null/undefined and return a
safe placeholder (e.g., 'N/A' or '-') instead of passing null through, and
update fmtPercent to guard against null before calling toFixed (return
placeholder when input is null). Reference functions: calculateChange, classify,
fmtDelta, fmtPercent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8e30b9d7-8862-4bb3-b317-0d1e84f48835
📒 Files selected for processing (3)
.github/workflows/pr.ymldocs/benchmarks.mddocs/build-system.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #116
Summary
Verification
github-scriptblock withnode --checkSummary by CodeRabbit
Refactor
Documentation