Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Refactor benchmark tools for statistical significance #7094
Affected core subsystem(s)
Description of change
I have been rather confused about the benchmark suite and I don't think it is as user friendly as the rest of nodecore. This PR attempt to remove most of the confusion I was facing when I started using it. Primarily it:
The specifics are documented in the commit messages. Please also see the the new README as quite a lot have changed (be sure the to check my spelling!).
Note that some benchmark takes a very long time to complete, e.g.
Motivation (long story): I wanted to benchmark the effect of some
I then had to search for the tools myself and discovered the large amount of benchmarks files which where not put into categorized directories. I assumed they where somehow extra significant, but in reality they just appear to be unused. These files are now removed.
After discovering the compare tool, which has the cli API
I was confused about what the
At this point I wanted to do a rewrite of the benchmark tools (not the benchmarks themself) and changed a few other things in the process as well. - I'm a mathematician so I care a lot about statistical significance :)
@mscdex Thanks for the suggestions, I will update the documentation tomorrow.
@mcollina It runs each the benchmark a given number of times (
I think the compare documentation is fairly clear on this. But do tell me how I can improve it.
Thanks for the review. Landed in ee2843b edbed3f 0f9bfaa f3463cf3061931b5c94ba9c753c1d75ee4d2b712 1f64ceba89a074f9e23196d019d56f00cdd4577a 01fbf656a3874d189cadeced08266a26ea526491 de9b44c0889d2264436277848762f1ebf868aa57 6e745d7a7586b12b894537192726bf2b999a456d 693e7be399e4c0964b5bbceaee6e8326c7c02a42
Thanks for the quick eye. I have force pushed and updated the PR. I wish I knew how it happened.
Well, yeah, I’ve had the, ahem, pleasure of breaking master by not having run CI again before landing myself in the recent past. :)
Anyway, CI looked good before it went all 502 (FreeBSD failure is unrelated and only the Windows tests were remaining), I’d say you can land this. Thanks!