Companion to the "Minimum Times Tend to Mislead When Benchmarking" blogpost
This repository hosts a simple experiment to show the dangers of using the minimum time of a benchmark as a measure of its performance.
Running the experiment
To run the experiment you need to have the following installed:
cargoinstalled (any recent version should do).
python. By default,
python-2.7will be used, though any
3.*version should work as well. Override the default choice by setting the
PYTHONenvironment variable e.g.
To run the experiment use the
$ make times
By default each experiment is run 100,000 times: on a fast machine this will
approximately 36 hours to run. You can change this value by setting the
environment variable. For example, to get a short run of the entire experiment
you might run things as follows:
$ RUNS=1000 make times
If you also want to generate graphs you will need to have the following installed:
seaborn(installed for whatever version of Python you are using)
make will run the experiment (if it has not been run) and then
generate graphs. If you have existing data (i.e. you have the files
linear_search_times in this
directory) then running
make will just generate SVG and PDF graphs without
running the experiment.