Permalink
Browse files

Updated readme and cabal for v0.2.0

  • Loading branch information...
1 parent 929162e commit 5382e0b0459df822568d85e8cc196b8f1de52fe8 @dmpots committed Nov 9, 2010
Showing with 136 additions and 34 deletions.
  1. +108 −19 README.md
  2. +28 −15 fibon.cabal
View
127 README.md
@@ -8,24 +8,39 @@ Fibon in a Flash
Introduction
===================================================================
-Fibon is a set of tools for running and analysing benchmarks in
-Haskell. Most importantly, it includes a set of [default benchmarks][3]
-taken from the [Hackage][1] open source repository.
+Fibon is a set of tools for running and analyzing benchmark programs in
+Haskell. Most importantly, it includes an optional set of new [benchmarks][2]
+including many programs taken from the [Hackage][1] open source repository.
+
+Fibon is a pure Haskell framework for running and analyzing benchmarks. Cabal
+is used for building the benchmarks, and the benchmark harness, configuration
+files, and benchmark descriptions are all written in Haskell. The benchmark
+descriptions and run configurations are all statically compiled into the
+benchmark runner to ensure that configuration errors are found at compile
+time.
+
+The Fibon tools are not tied to any compiler infrastructure and can build
+benchmarks using any compiler supported by cabal. However, there are some
+extra features available when using GHC to build the benchmarks:
+
+ * Support in config files for inplace GHC HEAD builds
+ * Support in `fibon-run` for collecting GC stats from GHC compiled programs
+ * Support in `fibon-analyse` for reading GC stats from Fibon result files
Benchmarks
------------------
-Fibon makes it easy to use either the fibon benchmarks or your own
+Fibon makes it easy to use either the Fibon benchmarks or your own
set of benchmarks. Benchmarks are stored in the
`benchmarks/Fibon/Benchmarks` directory. This directory is setup as
-a [git submodule][2] which means you can easily grab the standard
+a [git submodule][3] which means you can easily grab the standard
suite or use a suite kept under your own source control.
The default suite of benchmarks is stored in the
-[fibon-benchmarks][3] repository on Github.
+[fibon-benchmarks][2] repository on github.
Benchmark Groups
------------------
-Benchamarks named and organized into groups based on the filesystem
+Benchmarks are named and organized into groups based on the filesystem
organization. For example, a benchmark in the directory
`benchmarks/Fibon/Benchmarks/Hackage/Agum` will have the name `Agum`
an be in the benchmark group `Hackage`.
@@ -36,13 +51,14 @@ The fibon package builds three tools:
1. `fibon-run` - runs the benchmarks
2. `fibon-analyze` - analyzes the results of a run
-3. `fibon-init` - utilty used when adding new benchmarks
+3. `fibon-init` - utility used when adding new benchmarks
Size and Tune
------------------
-Fibon benchmarks can be run with two different input sizes: `Test` and
-`Ref`. The `Test` size is useful to make sure that a benchmark can
-run successsfully, but will not give meaningful timings.
+Fibon benchmarks can be run with two different input sizes: `Test` and `Ref`.
+The `Test` size is useful to make sure that a benchmark can run successfully,
+but will not give meaningful timings. The `Ref` size should be used when
+reporting results.
Fibon benchmarks can be run under two different tune settings (e.g.
compiler optimization settings). The `Base` and `Peak` settings can
@@ -51,30 +67,32 @@ be configured anyway you want to make the desired comparison.
Directory Structure
--------------------
Source directories
+
./benchmarks -- benchmark code
./config -- config files
./lib -- common files used by several executables
./tools -- source code for executables
Working directories
+
./log -- logging output from benchmark runs
./run -- working directory for benchmark runs
Getting the Benchmarks
===================================================================
The benchmarks are kept in a separate repository as a git
-submodule. You can get the fibon benchmarks by updating the
-submodule from within your fibon working directory
+submodule. You can get the Fibon benchmarks by updating the
+submodule from within your Fibon working directory
$ git submodule update --init benchmarks
-This will checkout the benchmarks from the [fibon-benchmarks][3]
+This will checkout the benchmarks from the [fibon-benchmarks][2]
repository and place them in your working copy.
Running Benchmarks
===================================================================
The available benchmarks and configurations are discovered when the
-fibon package is configured. Benchmarks are searched for in the
+Fibon package is configured. Benchmarks are searched for in the
`benchmarks/Fibon/Benchmarks` directory and configuration files are
searched for in the `config` directory. If a configuration file or
benchmark is added, you will need to re-run `cabal configure` to
@@ -99,7 +117,7 @@ groups, sizes, and tune settings as described below.
Running
---------------
-Benchamarks are run with the `fibon-run` tool. Running `fibon-run`
+Benchmarks are run with the `fibon-run` tool. Running `fibon-run`
with no arguments will use the default config file. An alternate
config file can be specified with the `-c` flag. Also, you can give
a list of benchmarks or groups to run on the command line. Use
@@ -119,6 +137,7 @@ Benchmarks can be analyzed by the `fibon-analyse` tool.
$ fibon-analyse log/000.default.RESULTS
or
+
$ fibon-analyse log/000.default.RESULTS.SHOW
The binary results (`.RESULT`) file is much faster to parse. It
@@ -128,7 +147,77 @@ parsed by using the `read` function.
Adding New Benchmarks
===================================================================
-TODO
+
+New benchmarks are added by putting the appropriate files in the
+`benchmarks/Fibon/Benchmarks` directory. Each folder in this directory
+represents a benchmark group. The benchmarks and groups are found at
+configuration time (i.e. when running `cabal configure` for the fibon
+package). You can exclude a benchmark or a group by prefixing the name with
+and underscore (`_`).
+
+To add a new benchmark create a new folder in a benchmark group. If the
+benchmark program has been cabalized, you can typically just do a
+`cabal unpack` of the benchmark. The benchmark folder must contain:
+
+ 1. A cabal file describing how to build the benchmark
+ 2. A benchmark description for Fibon stored in `Fibon/Instance.hs`
+
+The `fibon-init` tool will read a cabal file from the current directory and generate the Fibon subfolder and a stub `Instance.hs` file.
+
+The `Fibon` subfolder of a benchmark contains all of the data that Fibon needs
+to build and execute the benchmark. The benchmark instance file describes any
+requried build flags, inputs, and outputs for the benchmark. It is a standard
+Haskell module that must export the `mkInstance` function. The `mkInstance`
+function takes a benchmark size and returns a `BenchmarkInstance` structure.
+An example instance file is show below.
+
+ module Fibon.Benchmarks.Hackage.Bzlib.Fibon.Instance(
+ mkInstance
+ )
+ where
+ import Fibon.BenchmarkInstance
+
+ sharedConfig = BenchmarkInstance {
+ flagConfig = FlagConfig {
+ configureFlags = []
+ , buildFlags = []
+ , runFlags = []
+ }
+ , stdinInput = Nothing
+ , output = []
+ , exeName = "hsbzip"
+ }
+ flgCfg = flagConfig sharedConfig
+
+ mkInstance Test = sharedConfig {
+ flagConfig = flgCfg {
+ runFlags = ["bzlib.cabal.bz2"]
+ }
+ , output = [(OutputFile "bzlib.cabal.bz2.roundtrip",
+ Diff "bzlib.cabal.bz2")]
+ }
+ mkInstance Ref = sharedConfig {
+ flagConfig = flgCfg {
+ runFlags = ["mito.aa.bz2"]
+ }
+ , output = [(OutputFile "mito.aa.bz2.roundtrip",
+ Diff "mito.aa.bz2")]
+ }
+
+The input and expected output data should also be stored in the `Fibon`
+subdirectory of the benchmark. When the benchmark is run, the contents of the
+input and output directories for the benchmark size will be copied to the
+working directory where the benchmark is run. There can also be an `all`
+directory which whose data will be copied for all input sizes. None of the
+data directories are required, but if they exist they must be organized like
+this:
+
+ data/all/input
+ data/all/output
+ data/ref/input
+ data/ref/output
+ data/test/input
+ data/test/output
Benchmark Notes
===================================================================
@@ -147,5 +236,5 @@ Ghc610
Think it is just some kind of floating point wibbles.
[1]: http://hackage.haskell.org
-[2]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
-[3]: http://github.com/dmpots/fibon-benchmarks
+[2]: http://github.com/dmpots/fibon-benchmarks
+[3]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
View
@@ -1,22 +1,34 @@
-- The name of the package.
Name: fibon
-Version: 0.1.0
-Synopsis: A reworking of the classic nofib benchmark suite
+Version: 0.2.0
+Synopsis: Tools for running and analyzing Haskell benchmarks
Description:
Fibon is a set of tools for running and analyzing benchmark programs.
The fibon package contains the tools for benchmarking, but not the
benchmarks themselves. The package will build, but will not have any
- benchmarks to run. A set of benchmarks can be found at
+ benchmarks to run. A set of benchmarks can be found in the github repo
+ .
+ <http://github.com/dmpots/fibon-benchmarks>
+ .
+ Fibon is a pure Haskell framework for running and analyzing benchmarks. Cabal
+ is used for building the benchmarks, and the benchmark harness, configuration
+ files, and benchmark descriptions are all written in Haskell. The benchmark
+ descriptions and run configurations are all statically compiled into the
+ benchmark runner to ensure that configuration errors are found at compile
+ time.
+ .
+ The Fibon tools are not tied to any compiler infrastructure and can build
+ benchmarks using any compiler supported by cabal. However, there are some
+ extra features available when using GHC to build the benchmarks:
- http://github.com/dmpots/fibon-benchmarks
-
- Installing the cabal package will get you the following tools
-
- * fibon - runs the benchmarks
- * fibon-analyse - analyses the results of a benchmark run
- * fibon-init - generate a benchmark description from a cabal file
-
- See http://github.com/dmpots/fibon/wiki for more details
+ * Support in config files for inplace GHC HEAD builds
+
+ * Support in `fibon-run` for collecting GC stats from GHC compiled programs
+
+ * Support in `fibon-analyse` for reading GC stats from Fibon result files
+
+ .
+ For more details see the Fibon wiki: <http://github.com/dmpots/fibon/wiki>
License: BSD3
License-file: LICENSE
Author: David M Peixotto
@@ -28,7 +40,7 @@ Bug-reports: http://github.com/dmpots/fibon/issues
Build-type: Custom
Cabal-version: >=1.8
-Extra-source-files: README
+Extra-source-files: README.md
FindBench.hs
FindConfig.hs
lib/Fibon/BenchmarkInstance.hs
@@ -37,8 +49,8 @@ Extra-source-files: README
lib/Fibon/InputSize.hs
lib/Fibon/Result.hs
lib/Fibon/Timeout.hs
- tools/fibon-analyse/Fibon/Analyse/AnalysisRoutines.hs
tools/fibon-analyse/Fibon/Analyse/Analysis.hs
+ tools/fibon-analyse/Fibon/Analyse/AnalysisRoutines.hs
tools/fibon-analyse/Fibon/Analyse/CommandLine.hs
tools/fibon-analyse/Fibon/Analyse/ExtraStats/GhcStats.hs
tools/fibon-analyse/Fibon/Analyse/ExtraStats.hs
@@ -47,6 +59,7 @@ Extra-source-files: README
tools/fibon-analyse/Fibon/Analyse/Output.hs
tools/fibon-analyse/Fibon/Analyse/Parse.hs
tools/fibon-analyse/Fibon/Analyse/Result.hs
+ tools/fibon-analyse/Fibon/Analyse/Statistics.hs
tools/fibon-analyse/Fibon/Analyse/Tables.hs
tools/fibon-init/Main.hs
tools/fibon-run/Fibon/Run/Actions.hs
@@ -68,7 +81,7 @@ source-repository head
source-repository this
type: git
location: git://github.com/dmpots/fibon.git
- tag: v0.1.0
+ tag: v0.2.0
Flag analyse
description: Build the fibon-analyse program

0 comments on commit 5382e0b

Please sign in to comment.