General benchmarking framework. Especially good at parameter studies.
Haskell Shell Gnuplot Other
Permalink
Failed to load latest commit information.
analyze small fix for plotting script Jul 11, 2013
bench Refactor to move the main script into a library May 21, 2013
bin Add missing file. Bump stack.yaml up to LTS 3 and build hsbencher-gra… Nov 12, 2015
cat-csv Add cat-csv here to keep it together with the hsbencher stuff May 31, 2016
deps Fix warnings and wipe dynamic fields that would normally be populated… May 31, 2016
hsbencher-codespeed Various tweaks and version bumps for GHC 7.10 Apr 26, 2015
hsbencher-fusion Add another field: RSQR_TIME_LOW May 31, 2016
hsbencher-graph Add --inverse option and bump to 0.13 Jul 8, 2017
hsbencher Fixes to previous Jun 29, 2017
old Also file away hsbencher-analytics Apr 11, 2017
.build_tests.sh Add a separate stack-nightly.yaml Jun 2, 2016
.gitignore minor: add to git ignor Aug 19, 2015
.gitmodules Switch criterion over to https May 31, 2016
.jenkins_script.sh Refactor scripts to share between travis and jenkins Jun 2, 2016
.run_tests.sh Refactor scripts to share between travis and jenkins Jun 2, 2016
.travis.yml Refactor scripts to share between travis and jenkins Jun 2, 2016
DEVLOG.md Make a note about problematic stack.yaml behavior and make the nightl… Dec 23, 2015
README.md Update README.md Jun 3, 2016
TODO.md Add a configuration option to swap flags and regular cmd line argumen… May 31, 2013
example Move examples dir Apr 27, 2014
shell.nix Bump to 1.34 - add hacky feature for ingesting criterion output Jun 29, 2017
stack-lts-2.22.yaml Add a separate yaml file for legacy lts 2.22 Jun 1, 2016
stack-nightly.yaml Add a separate stack-nightly.yaml Jun 2, 2016
stack.yaml Bump to 1.34 - add hacky feature for ingesting criterion output Jun 29, 2017

README.md

HSBencher: A flexible benchmarking framework

Build/test Status:

  • Travis: Build Status
  • Jenkins: Build Status

Overview

See hsbencher.cabal for a general overview. Here are a few useful facts:

  • The hsbencher package is for describing benchmark configuration spaces, launching jobs, and collecting data for them
  • Jobs can take many forms. Some included protocols are described below, but you can always add your own BuildMethod.
  • Other packages like hsbencher-fusion and hsbencher-codespeed provide additional backends for uploading benchmark data to network destinations.

Below we describe some of the rules benchmarks must follow. But you will find much more general documentation on the project's Wiki.

Protocols for benchmarks to follow

All benchmarks, via all BuildMethods

Benchmarks using any BuildMethod, including BuildMethods added by the end user obey the following conventions:

  • Timing -- complete process runtime is used by default, this can be overridden by having the benchmark print out a line such as SELFTIMED 3.3 on stdout , which would indicate a 3.3 second runtime.

  • Coming soon -- compile time timing and multi-phase timing (e.g. for Accelerate)

  • "shortrun" -- all benchmarks should run and do SOMETHING even if given no runtime arguments. The convention is for quick tests (correctness, not performance) to run in this way. HSBencher supports a --shortrun mode that enables this. Note that it still passes in environment variables and "parameters", it elides only the cmdargs field of the Benchmark record.

Benchmarks using the builtin 'make' BuildMethod

  • make should build the benchmark.
  • make run should run the benchmark
  • compile time arguments are provided with make COMPILE_ARGS='...'
  • runtime arguments are provided with make run RUN_ARGS='...'

Benchmarks using the builtin cabal BuildMethod

  • One .cabal file should be contained in the directory.
  • Either the directory itself, or a file within the directory can be the benchmark "target".
  • ONE executable target should be built when cabal install or cabal-dev install is invoked.
  • compile time arguments are formatted for cabal-install
  • runtime arguments are provided to the resulting executable, raw (i.e. you need to include +RTS -RTS yourself)

Benchmarks using the builtin ghc BuildMethod

  • A single .hs file should be specified as the build target
  • It should build by running ghc --make on the target file; any include directories beyond the one containing the target file must be added explicitly (as CompileParam's)
  • compile time arguments are formatted for ghc command line
  • runtime arguments are provided to the resulting executable, raw (i.e. you need to include +RTS -RTS yourself)