Compare the execution runtimes of different implementations of Constraint Handling Rules.
The following systems with CHR support are compared. With CHR(X) we denote a CHR system as extension of language X.
- CHR(Prolog): SWI-Prolog, using KU Leuven CHR
- CHR(C): CCHR
- CHR(Java): JCHR
- CHR(JavaScript): CHR.js
In addition to these systems we provide native C and JavaScript implementations of most of the given problems.
The test cases are based on the paper "CCHR: the fastest CHR Implementation, in C." by Wuille, Schrijvers and Demoen (2007). The example problems are:
- fib: The bottom-up calculation of the Fibonacci numbers.
- gcd: The calculation of the greatest common divisor of two integers, based on the subtraction-based Euclidean algorithm.
- leq: A constraint solver for less-equal constraints between variables.
- primes: An implementation of the Sieve of Eratosthenes to generate prime numbers.
- ram: A simulator of a Random Access Machine.
- tak: Implementation of the Takeuchi function.
Due to different features of the examined CHR systems not all tests have been implemented for all systems.
The Makefile
contains a large number of targets. For every system (swi
, jchr
, cchr
, chrjs
and native c
) there are several sub-targets, in particular:
system.preinstall
: Installation of dependencies and benchmark setup, for example the creation of temporary directories.system.install
: Installs the actual system.system.prepare
: Preparation tasks for the benchmarks. Usually this includes the compilation of the test source files, for example compile*.jchr
files for JCHR.system.clean
: Target to delete temporary directories and files. This should be called before the benchmark is executed.system.test
: Runs each test once to check if it is executable. This will generally create no output. The tests have been passed if no error occurs.system.bench
: Executes the benchmarks for this system.
Apart from these tasks there are further, system dependent sub-targets, for example to benchmark only a single system and single test case.
To install all given systems and prepare the benchmarks, run these two commands:
$ sudo make install
$ sudo make prepare
To start the benchmarks simply call make bench
:
$ make bench
./bench.pl
## bench=leq
### sys=swi
swi/leq:1 (0,0.00186556577682495)*1696 exp
swi/leq:2 (0,0.00171436494731004)*1803 exp
swi/leq:3 (4.44119496322037e-05,0.00171436494731004)*1766 exp
...
There is no order in which systems and problems are tested, so leq/swi
does not need to be the first.
Because the benchmark for all systems will due several hours, make bench.save
should be preferred. It will create a bench.out
file with the benchmark results.
The bench.out
file can be used to create a plot with the benchmark results:
make plot
This will create the /plots
directory containing a PDF for each problem, for example /plots/bench-gcd-col.pdf
.
If you use the CHR Benchmarks in a scientific publication, please cite it as described in CITATION
.