Pbench: a program benchmarking toolkit
pbench toolkit is a software framework whose purpose is
to help programmers benchmark their programs, with particular support
for parallel programs. The toolkit provides two command-line tools,
pplot, that are useful for small, explorative
benchmarking scenarios. It also provides an OCaml library that is
useful for writing sophisticated performance evaluations of one or
more benchmark programs, and for generating polished tables and plots
ready for incorporation in research papers.
pruntool organizes and executes measured runs of client-supplied benchmark programs. Crashes and timeouts of benchmark programs are handled gracefully by
pplottool generates various human-readable output from the data collected by
prunexperiments. Forms of output include latex tables, R bar and scatter plots, etc. Data that is output by
pplotcan be lightly processed, for example, to show the mean run time of a number of runs. Moreover,
pplotcan also render error bars.
The OCaml library provided by
pbenchassists experimentalists who need fine-grained tuning of the runs and plots, and who want to make their experimental evaluations repeatable by other experimenters. The distribution includes example scripts using the lbirary.
The pbench plot generator requires that recent versions of
R and LaTeX be installed on the
host machine. When pplot generates new plots, all intermediate
R and latex sources that were generated by pplot are written
_results folder. It is not hard to customize the
look of the plots by editing these files individually.
Conventions for benchmark programs
The benchmark programs themselves can be written in any language. The only requirement is that the benchmark programs honor a certain format for receiving command-line arguments and printing measurements.
The program must receive arguments as lists of key-value pairs. The format must be as follows:
-key1 value1 -key2 value2 ...
Flag arguments of the form
--flag are also allowed.
Special characters such as commas and dashes must not appear in the strings of the keys and values.
The program must print measurements to
stdout in lists
of key-value pairs in the following format:
key1 value1 key2 value2 ...
Keys and values may be separated by one or more spaces or by tabs.
Output that deviates from this format cannot be parsed by
Example: Fibonacci benchmark program
The following example command-line program honors our conventions.
$ make -C examples/basic fib $ examples/basic/fib -algo recursive -n 39 result 102334155 exectime 0.495086
The command-line tools
The documentation of the
prun tool can be found
prun.md and that of the
pplot tool in
Writing custom benchmarking scripts
We have created an example script to demonstrate basic use of our library. The script (and the rest of our library) is written in ocaml. At least an intermediate level of expertise in ocaml is essential to write custom scripts.
The example script is named
example.ml and can be found
in the top-level folder of the pbench source tree.
The associated file named
example.md provides documentation
of the command-line usage of the script. Although the
source of the document is suitable for reading, the document
can alternatively be rendered and read as a PDF file by the
To build the ocaml script, namely
example.byte, make the
following build command.
The documentation explains the usage of the program.
For more advanced examples, see