Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Randomize execution order of benchmarks #7

Closed
smarr opened this issue Feb 28, 2013 · 1 comment
Closed

Randomize execution order of benchmarks #7

smarr opened this issue Feb 28, 2013 · 1 comment
Labels

Comments

@smarr
Copy link
Owner

smarr commented Feb 28, 2013

To avoid systematic bias that might be caused by operating system caches or hardware/memory properties, the execution order of benchmark runs should be randomized.

This has the additional benefit of producing data points for more benchmarks and it is possible to see trends of the results earlier for long running sets of benchmarks.

There needs to be an option to suppress the randomization, of course.
Also, we should have an estimate of what that means in terms of memory usage when many microbenchmarks are executed (need to keep the data in memory to calculated confidence interval, etc...)

@smarr
Copy link
Owner Author

smarr commented Jun 12, 2016

I think this has already been implemented, and is supported with:

  -s SCHEDULER, --scheduler=SCHEDULER
                        execution order of benchmarks: batch, round-robin,
                        random [default: batch]

@smarr smarr closed this as completed Jun 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant