Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
base fork: dmpots/fibon
base: cd9ae25043
...
head fork: dmpots/fibon
compare: 78d6f4214c
  • 2 commits
  • 3 files changed
  • 0 commit comments
  • 1 contributor
Commits on Oct 06, 2011
@dmpots Follow updates in benchmarks and config submodules
Benchmarks
> Fix Gf build for ghc 7.1
> Updated syb to version 0.3.2 for funsat benchmark

Config
> Added config for llvm-gcc GC comparisons
521653c
@dmpots Updated README with how to avoid long running times
Had questions from several people on why the benchmarks were
running so slow. The main reason was the switch to longer
running times for the Ref size without updating the config
files to account for the increased time.

Added a FAQ section that describes options to make
benchmarking faster.
78d6f42
Showing with 131 additions and 31 deletions.
  1. +129 −29 README.md
  2. +1 −1  benchmarks/Fibon/Benchmarks
  3. +1 −1  config
View
158 README.md
@@ -1,5 +1,6 @@
Fibon in a Flash
===================================================================
+
$ git clone git://github.com/dmpots/fibon.git
$ cd fibon
$ git submodule update --init benchmarks
@@ -8,6 +9,7 @@ Fibon in a Flash
Introduction
===================================================================
+
Fibon is a set of tools for running and analyzing benchmark programs in
Haskell. Most importantly, it includes an optional set of new [benchmarks][2]
including many programs taken from the [Hackage][1] open source repository.
@@ -45,25 +47,31 @@ organization. For example, a benchmark in the directory
`benchmarks/Fibon/Benchmarks/Hackage/Agum` will have the name `Agum`
an be in the benchmark group `Hackage`.
-Executables
-------------------
-The fibon package builds three tools:
-1. `fibon-run` - runs the benchmarks
-2. `fibon-analyze` - analyzes the results of a run
-3. `fibon-init` - utility used when adding new benchmarks
-
-Size and Tune
+Benchmark Size
------------------
-Fibon benchmarks can be run with two different input sizes: `Test` and `Ref`.
-The `Test` size is useful to make sure that a benchmark can run successfully,
-but will not give meaningful timings. The `Ref` size should be used when
+Fibon benchmarks can be run with three different input sizes: `Test`, `Train`
+and `Ref`. The `Test` size is useful to make sure that a benchmark can run
+successfully, but will not give meaningful timings. The `Train` size can be
+used to make measurements and should run in less than a minute. The `Ref` size
+is adjusted to give timingins similar to the SPEC benchmarks which typically
+run in 10 - 30 minutes. When possible, the `Ref` size should be used when
reporting results.
+Benchmark Tune
+---------------
Fibon benchmarks can be run under two different tune settings (e.g.
compiler optimization settings). The `Base` and `Peak` settings can
be configured anyway you want to make the desired comparison.
+Framework Executables
+----------------------
+The fibon package builds three tools:
+
+1. `fibon-run` - runs the benchmarks
+2. `fibon-analyze` - analyzes the results of a run
+3. `fibon-init` - utility used when adding new benchmarks
+
Directory Structure
--------------------
Source directories
@@ -100,12 +108,14 @@ make them available to the fibon-run tool.
Configuration
---------------
-Fibon comes with a default configuration. The default configuration
-will run all benchmarks with the `Base` setting of `-O0` and a
-`Peak` setting of `-O2` on the `Ref` size. A configuration file can
-be used to specify more complicated configurations.
+Fibon comes with a default configuration. The default configuration will run
+all benchmarks with the `Base` setting of `-O0` and a `Peak` setting of `-O2`
+on the `Ref` size. This configuration can take quite a while to run, so it
+would be best to adjust the configuration to to suite your time constraints. A
+configuration file can be used to specify more complicated configurations.
You can get some example configuration by doing
+
$ git submodule update --init config
This will checkout a repository of config files. Note that currently
@@ -130,6 +140,7 @@ create four output files in the `log` directory.
2. `*.SUMMARY` - the mean runtimes of each benchmark
3. `*.RESULTS` - the full results in binary format (pass to `fibon-analyse`)
4. `*.RESULTS.SHOW` - the full results in text format (pass to `fibon-analyse`)
+5. `*.RESULTS.CONFIG` - the configuration values used to run the test
Analyzing Benchmark Results
===================================================================
@@ -162,7 +173,8 @@ benchmark program has been cabalized, you can typically just do a
1. A cabal file describing how to build the benchmark
2. A benchmark description for Fibon stored in `Fibon/Instance.hs`
-The `fibon-init` tool will read a cabal file from the current directory and generate the Fibon subfolder and a stub `Instance.hs` file.
+The `fibon-init` tool will read a cabal file from the current directory and
+generate the Fibon subfolder and a stub `Instance.hs` file.
The `Fibon` subfolder of a benchmark contains all of the data that Fibon needs
to build and execute the benchmark. The benchmark instance file describes any
@@ -219,22 +231,110 @@ this:
data/test/input
data/test/output
-Benchmark Notes
+FAQ
===================================================================
-Ghc612
- The Repa and Dph groups will not work properly.
-Ghc610
- The Repa and Dph groups will not work properly.
-
- ChameneosRedux
- Does not work with -O0. Gets "thread blocked indefinitely"
- exception
+Why do the benchmarks take so long to run?
+--------------------------------------------------------
+
+All my benchmark runs are timing out. What's going on?
+--------------------------------------------------------
+
+
+A while back I changed the running time of Ref size on the benchmarks to match
+more closely what I saw with the SPEC benchmarks, which was around 10 minutes.
+I didn't update all the config files to take that into account and that can
+make some benchmark runs take a long time.
+
+There are 5 ways to lower the overall run time:
+
+1. Run with smaller inputs
+2. Run fewer iterations
+3. Run with only one tune setting
+4. Limit benchmark execution time
+5. Run fewer benchmarks
+
+Some of these changes can be made in the config file or from the command line.
+See the [fibon-config][4] github repo for some example config files.
+
+### Run with smaller inputs ###
+
+Just like the SPEC benchmarks, the Fibon suite has both a "test", a "train",
+and a "ref" size. The "test" size is not likely to give a very accurate
+performance difference, but it runs very quickly so it can be useful for
+testing. The "train" size is a good intermediate value and each benchmark
+program should execute in less than a minute. The "ref" size will have longer
+benchmark runs where each program will run for 10-30 minutes per iteration.
+
+You can set the size from the config file:
+
+ sizeList = [Train]
+
+or on the command line:
+
+ $ fibon-run --size=Train
+
+### Run fewer iterations ###
+
+You can run each benchmark fewer times by specifying the iterations in the config file
+
+ iterations = 3
+
+or on the command line
+
+ $ fibon-run --iters=3
+
+### Run with only one tune setting ###
+
+I modeled Fibon after the SPEC benchmarks which have a notion of different
+"tune" settings: base and peak. The default config has base as `-O0` and peak
+as `-O2`. Having two tune settings means that every benchmark will be compiled
+twice and run at least twice. A benchmark compiled with `-O0` will run very
+slowly. You can run only one tune setting by setting it in the config file:
+
+ tuneList = [Peak]
+
+or on the command line:
+
+ $ fibon-run --tune=Peak
+
+
+### Limit benchmark execution time ###
+
+You can limit the execution time for the benchmarks. This can be an easy way to
+keep a benchmark from eating up all the execution time for the suite. You can
+set this in the config file. To limit execution time for all benchmarks, put it
+in the default builder section:
+
+ build :: ConfigBuilder
+ build ConfigTuneDefault ConfigBenchDefault = do
+ setTimeout $ Limit 0 0 15
+
+The constructor `Limit` takes ints for the maximum number of hours, minutes,
+and seconds. So `Limit 0 0 15` will limit all benchmark runs to 15 seconds. You
+can also set this on a per-benchmark or per-tune basis:
+
+ build (ConfigTuneDefault) (ConfigBench Cpsa) = do
+ -- give Cpsa a little longer
+ setTimeout $ Limit 0 0 30
+
+This currently only works to limit the actual running time of a benchmark, not the compile time.
+
+### Run fewer benchmarks ###
+
+You can run a subset of the full suite. Cutting out the benchmarks that take a
+long time to compile or run can make it complete much faster. You can set the
+run list in the configuration file
+
+ runList = [RunSingle agum, RunGroup Repa]
+
+or from the command line
+
+ $ fibon-run Agum Repa
- Mandelbrot
- The Test size gives different result, but the Ref size is ok.
- Think it is just some kind of floating point wibbles.
[1]: http://hackage.haskell.org
-[2]: http://github.com/dmpots/fibon-benchmarks
+[2]: https://github.com/dmpots/fibon-benchmarks
[3]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
+[4]: https://github.com/dmpots/fibon-config
+
2  benchmarks/Fibon/Benchmarks
@@ -1 +1 @@
-Subproject commit dd60a56377e94a0881f77afd9e4e2b621299e45e
+Subproject commit 3af6a2cbe3801e9533e8d95b411e50ed7b0cab44
2  config
@@ -1 +1 @@
-Subproject commit 68ca59018d43a6372f8e1478a427b4c7561011d9
+Subproject commit 0c0c4394462b47a54eafac4722099a72300a109b

No commit comments for this range

Something went wrong with that request. Please try again.