Support for comparing benchmarks against reference benchmarks #9

Merged
merged 2 commits into from Mar 7, 2012

Projects

None yet

2 participants

Contributor

Hi Bryan,

The goal of my vector-bytestring benchmark suite is to compare bytestring with vector-bytestring. It contains benchmarks for all functions from bytestring and all similar functions from vector-bytestring. What I want to know is how much faster vector-bytestring is compared to bytestring. So currently I run the benchmarks, manually load them into a spreadsheet to calculate how much faster (in %) the vector-bytestring functions are and finally do some post-processing in emacs to prettify the result.

I would like to automate this and came up with the following solution:

There's a new combinator:

bcompare :: [Benchmark] -> Benchmark

The first bench in the list (no matter how deep it's nested) is the reference benchmark. All other benchmarks are compared against that reference. The user can specify to write the comparisons to a CSV file using the -r (or --compare) command line flag. The CSV file uses the following format:

Reference,Name,% faster than the reference

where the % is currently printed without precision (%.0f).

Note that a bcompare can contain any Benchmark. So they can also contain bgroups and even other bcompares. The semantics of the latter are that:

  • all the benchmarks within the nested bcompare are compared against the reference of the parent
  • recursive step: all the benchmarks within the nested bcompare (except the first because it's the reference) are compared against the reference of the nested bcompare.

To use bcompare in vector-bytestring I only needed to make a tiny change!

basvandijk added some commits Jan 29, 2012
@basvandijk basvandijk Support for comparing benchmarks against reference benchmarks
There's a new combinator: bcompare :: [Benchmark] -> Benchmark
The first 'bench' in the list is the reference benchmark.
All other benchmarks are compared against that reference.
The user can specify to write the comparisons to a CSV file
using the -r (or --compare) command line flag.
The CSV file uses the following format:
Reference,Name,% faster than the reference
where the % is currently printed without precision (%.0f).
2b27700
@basvandijk basvandijk Fix documentation of bcompare a302a3e
@bos bos merged commit a302a3e into bos:master Mar 7, 2012
bos commented on 2b27700 Mar 7, 2012

Nice work, Bas! This is a good idea. Sorry for the delay - new baby, fatigue, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment