Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
After the release of 0.3.0, several folks commented that Criterion.rs' handling of long-running benchmarks is not great - it can take a very long time to measure these functions. That's true! It would be nice to improve on that. This is more of a tracking issue to describe the work required.
The biggest problem preventing this is an API one. I'd need to allow the user to specify whether they want each benchmark to use the slow- or fast-benchmark process, and (at the moment) I don't know what that should look like. Ultimately it would require a judgement call by the programmer and I'd prefer to try to avoid those where possible. One of the goals of Criterion.rs is to not require the benchmark author to understand statistics, but then how can they make an informed decision?
The analysis and measurement is relatively straightforward - since we can assume timing jitter is insignificant compared to the function itself, just run a smaller number of iterations and measure each one individually. That breaks all of the slope-based analysis so it would require new analysis code. Fortunately, the usual statistics (mean, std. dev., etc.) should all work more-or-less well, so that's probably OK, but it requires internal design changes and a bunch of work. It would probably also need different configuration settings than the fast-benchmark process.
I have an FFT benchmark that is parameterized over input size, ranging from several elements to several millions. The
Criterion is amazing for accurate numbers in the low end, but no matter what configuration parameters I provide, it seems to insist on running benches dozens of times, for benches taking many minutes, this takes hours to complete.
I would love for there to be a "run this bench exactly once" mode. I understand this does not give any statistical assurances, but for minute long functions this really doesn't matter much.