Permalink
Browse files

Updated README with how to avoid long running times

Had questions from several people on why the benchmarks were
running so slow. The main reason was the switch to longer
running times for the Ref size without updating the config
files to account for the increased time.

Added a FAQ section that describes options to make
benchmarking faster.
  • Loading branch information...
dmpots committed Oct 6, 2011
1 parent 521653c commit 78d6f4214c84757ca32278e0dcab3a8ae80b2c84
Showing with 129 additions and 29 deletions.
  1. +129 −29 README.md
View
158 README.md
@@ -1,5 +1,6 @@
Fibon in a Flash
===================================================================
+
$ git clone git://github.com/dmpots/fibon.git
$ cd fibon
$ git submodule update --init benchmarks
@@ -8,6 +9,7 @@ Fibon in a Flash
Introduction
===================================================================
+
Fibon is a set of tools for running and analyzing benchmark programs in
Haskell. Most importantly, it includes an optional set of new [benchmarks][2]
including many programs taken from the [Hackage][1] open source repository.
@@ -45,25 +47,31 @@ organization. For example, a benchmark in the directory
`benchmarks/Fibon/Benchmarks/Hackage/Agum` will have the name `Agum`
an be in the benchmark group `Hackage`.
-Executables
-------------------
-The fibon package builds three tools:
-1. `fibon-run` - runs the benchmarks
-2. `fibon-analyze` - analyzes the results of a run
-3. `fibon-init` - utility used when adding new benchmarks
-
-Size and Tune
+Benchmark Size
------------------
-Fibon benchmarks can be run with two different input sizes: `Test` and `Ref`.
-The `Test` size is useful to make sure that a benchmark can run successfully,
-but will not give meaningful timings. The `Ref` size should be used when
+Fibon benchmarks can be run with three different input sizes: `Test`, `Train`
+and `Ref`. The `Test` size is useful to make sure that a benchmark can run
+successfully, but will not give meaningful timings. The `Train` size can be
+used to make measurements and should run in less than a minute. The `Ref` size
+is adjusted to give timingins similar to the SPEC benchmarks which typically
+run in 10 - 30 minutes. When possible, the `Ref` size should be used when
reporting results.
+Benchmark Tune
+---------------
Fibon benchmarks can be run under two different tune settings (e.g.
compiler optimization settings). The `Base` and `Peak` settings can
be configured anyway you want to make the desired comparison.
+Framework Executables
+----------------------
+The fibon package builds three tools:
+
+1. `fibon-run` - runs the benchmarks
+2. `fibon-analyze` - analyzes the results of a run
+3. `fibon-init` - utility used when adding new benchmarks
+
Directory Structure
--------------------
Source directories
@@ -100,12 +108,14 @@ make them available to the fibon-run tool.
Configuration
---------------
-Fibon comes with a default configuration. The default configuration
-will run all benchmarks with the `Base` setting of `-O0` and a
-`Peak` setting of `-O2` on the `Ref` size. A configuration file can
-be used to specify more complicated configurations.
+Fibon comes with a default configuration. The default configuration will run
+all benchmarks with the `Base` setting of `-O0` and a `Peak` setting of `-O2`
+on the `Ref` size. This configuration can take quite a while to run, so it
+would be best to adjust the configuration to to suite your time constraints. A
+configuration file can be used to specify more complicated configurations.
You can get some example configuration by doing
+
$ git submodule update --init config
This will checkout a repository of config files. Note that currently
@@ -130,6 +140,7 @@ create four output files in the `log` directory.
2. `*.SUMMARY` - the mean runtimes of each benchmark
3. `*.RESULTS` - the full results in binary format (pass to `fibon-analyse`)
4. `*.RESULTS.SHOW` - the full results in text format (pass to `fibon-analyse`)
+5. `*.RESULTS.CONFIG` - the configuration values used to run the test
Analyzing Benchmark Results
===================================================================
@@ -162,7 +173,8 @@ benchmark program has been cabalized, you can typically just do a
1. A cabal file describing how to build the benchmark
2. A benchmark description for Fibon stored in `Fibon/Instance.hs`
-The `fibon-init` tool will read a cabal file from the current directory and generate the Fibon subfolder and a stub `Instance.hs` file.
+The `fibon-init` tool will read a cabal file from the current directory and
+generate the Fibon subfolder and a stub `Instance.hs` file.
The `Fibon` subfolder of a benchmark contains all of the data that Fibon needs
to build and execute the benchmark. The benchmark instance file describes any
@@ -219,22 +231,110 @@ this:
data/test/input
data/test/output
-Benchmark Notes
+FAQ
===================================================================
-Ghc612
- The Repa and Dph groups will not work properly.
-Ghc610
- The Repa and Dph groups will not work properly.
-
- ChameneosRedux
- Does not work with -O0. Gets "thread blocked indefinitely"
- exception
+Why do the benchmarks take so long to run?
+--------------------------------------------------------
+
+All my benchmark runs are timing out. What's going on?
+--------------------------------------------------------
+
+
+A while back I changed the running time of Ref size on the benchmarks to match
+more closely what I saw with the SPEC benchmarks, which was around 10 minutes.
+I didn't update all the config files to take that into account and that can
+make some benchmark runs take a long time.
+
+There are 5 ways to lower the overall run time:
+
+1. Run with smaller inputs
+2. Run fewer iterations
+3. Run with only one tune setting
+4. Limit benchmark execution time
+5. Run fewer benchmarks
+
+Some of these changes can be made in the config file or from the command line.
+See the [fibon-config][4] github repo for some example config files.
+
+### Run with smaller inputs ###
+
+Just like the SPEC benchmarks, the Fibon suite has both a "test", a "train",
+and a "ref" size. The "test" size is not likely to give a very accurate
+performance difference, but it runs very quickly so it can be useful for
+testing. The "train" size is a good intermediate value and each benchmark
+program should execute in less than a minute. The "ref" size will have longer
+benchmark runs where each program will run for 10-30 minutes per iteration.
+
+You can set the size from the config file:
+
+ sizeList = [Train]
+
+or on the command line:
+
+ $ fibon-run --size=Train
+
+### Run fewer iterations ###
+
+You can run each benchmark fewer times by specifying the iterations in the config file
+
+ iterations = 3
+
+or on the command line
+
+ $ fibon-run --iters=3
+
+### Run with only one tune setting ###
+
+I modeled Fibon after the SPEC benchmarks which have a notion of different
+"tune" settings: base and peak. The default config has base as `-O0` and peak
+as `-O2`. Having two tune settings means that every benchmark will be compiled
+twice and run at least twice. A benchmark compiled with `-O0` will run very
+slowly. You can run only one tune setting by setting it in the config file:
+
+ tuneList = [Peak]
+
+or on the command line:
+
+ $ fibon-run --tune=Peak
+
+
+### Limit benchmark execution time ###
+
+You can limit the execution time for the benchmarks. This can be an easy way to
+keep a benchmark from eating up all the execution time for the suite. You can
+set this in the config file. To limit execution time for all benchmarks, put it
+in the default builder section:
+
+ build :: ConfigBuilder
+ build ConfigTuneDefault ConfigBenchDefault = do
+ setTimeout $ Limit 0 0 15
+
+The constructor `Limit` takes ints for the maximum number of hours, minutes,
+and seconds. So `Limit 0 0 15` will limit all benchmark runs to 15 seconds. You
+can also set this on a per-benchmark or per-tune basis:
+
+ build (ConfigTuneDefault) (ConfigBench Cpsa) = do
+ -- give Cpsa a little longer
+ setTimeout $ Limit 0 0 30
+
+This currently only works to limit the actual running time of a benchmark, not the compile time.
+
+### Run fewer benchmarks ###
+
+You can run a subset of the full suite. Cutting out the benchmarks that take a
+long time to compile or run can make it complete much faster. You can set the
+run list in the configuration file
+
+ runList = [RunSingle agum, RunGroup Repa]
+
+or from the command line
+
+ $ fibon-run Agum Repa
- Mandelbrot
- The Test size gives different result, but the Ref size is ok.
- Think it is just some kind of floating point wibbles.
[1]: http://hackage.haskell.org
-[2]: http://github.com/dmpots/fibon-benchmarks
+[2]: https://github.com/dmpots/fibon-benchmarks
[3]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
+[4]: https://github.com/dmpots/fibon-config
+

0 comments on commit 78d6f42

Please sign in to comment.