Skip to content
A simple experiment showing that using the minimum time of a benchmark can be misleading
Python Makefile Rust
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
binary_search
hashmap
linear_search
.gitignore
Makefile
README.md
graph.py
run.py

README.md

Companion to the "Minimum Times Tend to Mislead When Benchmarking" blogpost

This repository hosts a simple experiment to show the dangers of using the minimum time of a benchmark as a measure of its performance.

Running the experiment

To run the experiment you need to have the following installed:

  • rustc and cargo installed (any recent version should do).
  • python. By default, python-2.7 will be used, though any 3.* version should work as well. Override the default choice by setting the PYTHON environment variable e.g. PYTHON=python-3.6 make.

To run the experiment use the times target:

$ make times

By default each experiment is run 100,000 times: on a fast machine this will approximately 36 hours to run. You can change this value by setting the RUNS environment variable. For example, to get a short run of the entire experiment you might run things as follows:

$ RUNS=1000 make times

Generating graphs

If you also want to generate graphs you will need to have the following installed:

  • matplotlib and seaborn (installed for whatever version of Python you are using)
  • pdftocairo (part of poppler)
  • scour

Running make will run the experiment (if it has not been run) and then generate graphs. If you have existing data (i.e. you have the files binary_search_times, hashmap_times, linear_search_times in this directory) then running make will just generate SVG and PDF graphs without running the experiment.

You can’t perform that action at this time.