-
Notifications
You must be signed in to change notification settings - Fork 3
Increase default repetitions during codspeed walltime runs, and fix walltime stats computation #34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase default repetitions during codspeed walltime runs, and fix walltime stats computation #34
Conversation
01bad40 to
5af4562
Compare
CodSpeed Performance ReportMerging #34 will not alter performanceComparing Summary
Footnotes |
not-matthias
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good overall. Can we add some tests to ensure the logic works this time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the walltime benchmark data collection and statistics computation to support variable iterations per round. Instead of storing pre-aggregated statistics (mean, median, stdev) from Google Benchmark, it now collects raw per-round data (iterations and total time) and computes statistics based on per-iteration times.
Key changes:
- Changed
RawWalltimeBenchmarkstructure to store vectors of iterations and times per round instead of pre-computed aggregates - Modified statistics computation to calculate per-iteration times and derive all statistics from these values
- Increased default benchmark repetitions to 5 when
CODSPEED_WALLTIMEis enabled for better statistical reliability
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| core/include/codspeed.h | Updated RawWalltimeBenchmark struct to store vectors of per-round iterations and times instead of pre-aggregated statistics |
| core/src/walltime.cpp | Refactored statistics computation to work with raw per-round data, computing per-iteration statistics; added constructor to BenchmarkStats and made helper functions static |
| google_benchmark/src/benchmark.cc | Updated data collection to store per-round iterations and times; changed default repetitions to 5 for walltime mode; minor formatting and comment improvements |
| .github/workflows/ci.yml | Updated job step name to better reflect its purpose |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
adriencaccia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can disregard copilot's comments, since we are asserting the length of the array
…as an input The computation is now in line with that codspeed-rust does, the interface is ready for other frameworks that may different iterations count from one round to another.
This allows codspeed to have more rounds and improves default statistics, which would just be computed as 1 single element in the boxplot since we only have access to round times.
fa18cdf to
035fac4
Compare
035fac4 to
da312ec
Compare
Followup of #33, but the assumption made on what
getAdujstedRealTimewas doing were wrongWe're also changing the default number of repetitions (round), since we believe 1 is not a good default, as the resulting box plot only has a single element by default