fix: improve benchmark stability and adjust alert threshold#15
Merged
fix: improve benchmark stability and adjust alert threshold#15
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the benchmarks GitHub Actions workflow to reduce noisy benchmark alerts on PRs and improve benchmark run stability on shared runners.
Changes:
- Ignore docs/markdown-only changes for
pull_requestbenchmark runs. - Switch BenchmarkDotNet execution from a custom “short” config to
--job mediumfor more stable measurements. - Increase the benchmark regression alert threshold from 120% to 150%.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
55
to
+58
| --filter "*" | ||
| --exporters json | ||
| --join | ||
| --job short | ||
| --iterationCount 3 | ||
| --warmupCount 1 | ||
| --job medium |
There was a problem hiding this comment.
This step previously builds in a separate step, and dotnet run will build again by default. Consider adding --no-build to the dotnet run command to avoid redundant compilation (faster workflow and less noise for benchmark timing).
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Should get more stable benchmark results and no more false flags on PR merges. cc @PhenX
https://efnext.github.io/ExpressiveSharp/dev/bench/
Raised the alert-threshold to 150%. That's higher than I'd like it to be but the benchmarks are running on shared gh action runners with noisy neighbors (allegedly)