Find file History
Pull request Compare This branch is 6 commits ahead of kragen:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
bin
highlight
makefiles
README
license.txt
nanobench

README

	nanobench - The Computer Language Benchmarks Game

 Sep 2008 Isaac Gouy

 UPDATE July 2010

 - use BENCHER to measure programs of your own, see shootout/bencher      


 AUDIENCE
 
 - someone who wants to check how the benchmarks game measures programs


 MOTIVATION

 Too many problems getting the old minibench Perl script to run on a
 new Ubuntu x64 quadcore install - specifically with the GTOP wrapper.

 nanobench is written to create the data files, markup files and text
 files that have been presented through "The Computer Language Benchmarks
 Game" website over the past 4 years.


 REQUIREMENTS

 Linux (possibly maybe some other Unix flavour)
 Python 2.5
 gnu make
 cvs (possibly maybe)
 cmp
 diff
 ndiff		http://www.math.utah.edu/~beebe/software/ndiff/
 highlight	http://www.andre-simon.de/zip/download.html
 affinity (possibly maybe)	http://pypi.python.org/pypi/affinity/0.1.0

 
 BASIC ASSUMPTIONS
 
 Program source files are organized by 'benchmark' in different subdirectories,
 and program source files have the same name as the 'benchmark' distinguished
 by file extension, for example:

  [src directory]
     my_first_benchmark                         [subdirectory]
        my_first_benchmark.java                 [program source file]
        my_first_benchmark.gcc                  [program source file]
        my_first_benchmark.gcc-2.gcc            [program source file]

     my_second_benchmark                        [subdirectory]
        my_second_benchmark.python              [program source file]
        my_second_benchmark.gcc                 [program source file]
        my_second_benchmark.perl-2.perl         [program source file]


 LOCAL CONFIGURATION

 1) copy, rename and modify - /shootout/nanobench/makefiles/u32q.ini
 2) copy, rename and modify - /shootout/nanobench/makefiles/u32q.programs.Makefile

 Specifically,
  ROOT DIRS
   'root' is the directory where you want nanobench to put measurements and files.
   'src' is the directory that contains benchmark directories with source code.
   'nano' is the CVS directory /shootout/nanobench

  SWEEP DIRS
   Either remove dat_sweep, code_sweep, log_sweep from *.ini or set them to
   directories that exist - nanobench will not create them.

  'makefile' should be the path to your renamed *.programs.Makefile

 3) copy, rename and modify the bash script - /shootout/nanobench/nanobench


 Generally,
  filter the benchmarks you're interested in using ONLYDIRS
  filter the language implementations you're interested in using ONLY or IGNORE

  use your renamed *.programs.Makefile to prepare or build a program before the run.


 PROGRAM OUTPUT

  1) The code_sweep directory will contain the accumulated log files and
  'highlight' markup files shown on "The Computer Language Benchmarks Game" website.

  2) The dat_sweep directory (or the root directory) will contain the data.csv 
  & ndata.csv data files shown on "The Computer Language Benchmarks Game" website
  (they only have the fastest measurements from repeated runs); and a bzipped
  *-bulkdata.csv which contains ALL the measurements for programs which did not
  fail - so that's probably what you'll want to look at.

  3) for each source code subdirectory, a matching directory will be created
  under root, containing
   code subdirectory - symlinks to source files and 'highlight' markup files
   data subdirectory - compressed measurement files
   log subdirectory - log text files recording each program build and run
   tmp subdirectory - files needed to be build and measure the current program
   expected output files for various data input sizes *_out


 USAGE
 
  use the nanobench bash script to invoke nanobench with your particular *.ini file

  measurements are made when a source code file in a src subdirectory doesn't have a
  corresponding *.dat file or the *.dat file is older.

  re-measurement can be forced by deleting a particular *.dat file

  re-measurement can be forced for a particular source file extension
     ./nanobench --force java

  re-measurement can be forced for a particular benchmark
     ./nanobench --force binarytrees


 TESTDATA

 the location of the test data redirected to stdin for knucleotide, regexdna,
 and revcomp is given by the relative path in the *.ini file section [testdata]

 test data files of appropriate sizes should be generated using the fasta program
 and named according to benchmark and size, for example: 
 
  knucleotide-input50000.txt
  knucleotide-input500000.txt
  knucleotide-input5000000.txt

  regexdna-input50000.txt
  regexdna-input500000.txt
  regexdna-input5000000.txt