Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Handling of Long Benchmarks #320

Open
bheisler opened this issue Aug 26, 2019 · 1 comment

Comments

@bheisler
Copy link
Owner

@bheisler bheisler commented Aug 26, 2019

After the release of 0.3.0, several folks commented that Criterion.rs' handling of long-running benchmarks is not great - it can take a very long time to measure these functions. That's true! It would be nice to improve on that. This is more of a tracking issue to describe the work required.

The biggest problem preventing this is an API one. I'd need to allow the user to specify whether they want each benchmark to use the slow- or fast-benchmark process, and (at the moment) I don't know what that should look like. Ultimately it would require a judgement call by the programmer and I'd prefer to try to avoid those where possible. One of the goals of Criterion.rs is to not require the benchmark author to understand statistics, but then how can they make an informed decision?

The analysis and measurement is relatively straightforward - since we can assume timing jitter is insignificant compared to the function itself, just run a smaller number of iterations and measure each one individually. That breaks all of the slope-based analysis so it would require new analysis code. Fortunately, the usual statistics (mean, std. dev., etc.) should all work more-or-less well, so that's probably OK, but it requires internal design changes and a bunch of work. It would probably also need different configuration settings than the fast-benchmark process.

@Recmo

This comment has been minimized.

Copy link

@Recmo Recmo commented Oct 30, 2019

I have an FFT benchmark that is parameterized over input size, ranging from several elements to several millions. The n log n execution time ranges from a few nanoseconds to tens of minutes.

Criterion is amazing for accurate numbers in the low end, but no matter what configuration parameters I provide, it seems to insist on running benches dozens of times, for benches taking many minutes, this takes hours to complete.

I would love for there to be a "run this bench exactly once" mode. I understand this does not give any statistical assurances, but for minute long functions this really doesn't matter much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.