Skip to content
Benchmark and Regression Suite for Clojure
Clojure Scala Shell
Find file
Failed to load latest commit information.
archive
script Fix to script pull_nonca Jun 11, 2012
src add calling benchmark Jul 10, 2014
.gitignore First revision of baseline test structure, includes mandelbrot and th… Feb 28, 2012
CONTRIBUTING.md Add CONTRIBUTING.md May 20, 2014
README.markdown readme timing Feb 11, 2014
benchmarks.txt benchmarking approach: notes from Rich Oct 24, 2013
epl-v10.html initial commit Oct 26, 2010
pom.xml brand new knucleotide version ported from scala Apr 3, 2014

README.markdown

Benchmark (and regression) suite for Clojure Copyright (c) Rich Hickey.

License and CA same as Clojure. See epl-v10.html at the root of the project for details.

Getting Started

Requirements

Add the following to your ~/.m2/settings.xml file

<settings>
    <activeProfiles>
        <activeProfile>clojure-dev</activeProfile>
    </activeProfiles>

    <profiles>
        <profile>
            <id>clojure-dev</id>
            <activation> <activeByDefault>false</activeByDefault> </activation>
            <repositories>
              <repository>
                <id>clojars</id>
                <url>http://clojars.org/repo/</url>
              </repository>
            </repositories>
        </profile>
        ...
    </profiles>
    ...
</settings>

Building

git clone git@github.com:clojure/test.benchmark.git
cd test.benchmark
mvn test

Misc

# starts a swank server
mvn clojure:swank

Running a benchmark (e.g. alioth.thread-ring)

script/run alioth.thread-ring 1000

Timing a benchmark (e.g. alioth.fasta-redux)

time script/run alioth.fasta-redux 25000000 > /dev/null

real    0m2.389s
user    0m3.329s
sys 0m0.115s

In these results, alioth uses a measurement similar to the real timing.

Running a baseline test

Baseline tests execute a java-equivalent benchmark and compare execution times to the respective clojure benchmarks. The specifications for the baseline tests are in https://github.com/clojure/test.benchmark/blob/master/src/baseline/clojure/baseline/spec.clj

# this will pull down the non-CA repo into src/baseline and build
# Currently some Java programs require JDK 1.7 for Fork/Join library.
mvn compile -Pbaseline

# afterwards you can execute baseline tests as follows

# default, run all baseline tests 3x, sample and report
script/run baseline.exec

# run all baseline tests 10x, sample and report
script/run baseline.exec 10

# run mandelbrot and thread-ring baseline tests once and report
script/run baseline.exec 1 mandelbrot thread-ring
({:failures :none,
  :id "mandelbrot",
  :metrics {:runtime {:baseline 9451.773698, :target 10316.261803}},
  :tollerances {:runtime 0.25},
  :sample-size 1}
 {:failures :none,
  :id "thread-ring",
  :metrics {:runtime {:baseline 4308.804918, :target 309.521524}},
  :tollerances {:runtime -9.0},
  :sample-size 1})

TODO: hook baseline test execution into maven test cycle and generate a jenkins-consumable report

Contributing

There are a number of completed benchmarks, but there is a lot more left to do. Andy Fingerhut has a suit of benchmarks tailored for Clojure 1.2 and need updated or redone for Clojure 1.3 (https://github.com/jafingerhut/clojure-benchmarks).

A good general approach is to examine the fastest alioth implementations, usually Java or C, and write a Clojure port. Fast small programs in any language are also worthy of examination. These benchmarks should demonstrate how to write idiomatic high-performance Clojure.

Long term goal is a performance regression test suit ...

Alioth Benchmark site is here: http://shootout.alioth.debian.org/

Work on the benchmarks is tracked here: http://dev.clojure.org/jira/browse/TBENCH

Something went wrong with that request. Please try again.