Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
670 lines (476 sloc) 16.2 KB
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en-us"><head>
<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>Clojure whole program benchmarks</title>
<meta content="Andy Fingerhut" name="author"></head><body>[<a href="Clojure-benchmarks.html">benchmarks main page</a>] [whole program benchmarks] [<a href="Clojure-expression-benchmarks.html">expression benchmarks</a>] [<a href="Clojure-version-history.html">Clojure version history</a>] [<a href="Hardware-and-software-used-for-Clojure-benchmarks.html">hardware and software used</a>]
<h1><a class="mozTocH1" name="mozTocId411050"></a>Clojure whole program benchmarks</h1>For each program, the tables below include:<br>
<ul>
<li>a description of what the program should do.</li>
<li>a link to the source code of the Clojure and Java programs.</li>
<li>the command line arguments used.</li>
<li>any versions of Clojure that were not measured, and why.</li>
<li>some
statistics about the amount of input read, output written, and in the
last table the number of function calls performed during an execution
of the program.</li>
</ul>
All Clojure programs were compiled AOT (Ahead Of Time) before they were
executed.&nbsp; The compilation times are not included in the run time
measurements.&nbsp; What is included in the run time measurements are:<br>
<ul>
<li>the JVM startup time</li>
<li>Clojure startup time</li>
<li>"JIT warmup time"</li>
<li>time to read the input file, if any</li>
<li>time to write the output file</li>
</ul>
In other words, what you would get from the elapsed time measurement printed when you run the command:<br>
<br>
&nbsp;&nbsp;&nbsp; /usr/bin/time java -server -Xmixed -XX:+TieredCompilation -XX:+AggressiveOpts&nbsp; <span style="font-style: italic;">extra-jvm-options</span>&nbsp; -classpath <span style="font-style: italic;">classpath</span>&nbsp; <span style="font-style: italic;">class_name</span>&nbsp; <span style="font-style: italic;">additional_args</span>&nbsp; &gt; output_file<br>
<br>Each combination of (benchmark, Clojure version, platform) was run 3
times, and only the smallest elapsed time among them is shown in the
graph.&nbsp; By "platform" I mean the JVM, OS, and hardware running
beneath Clojure.<br>
<br>
If you are interested in what the JVM and Clojure startup times are by
themselves, see the "hello" benchmark, which does nothing but print
"Hello, world!".&nbsp; This ranges from about 1 to 2 seconds depending
upon the Clojure version and JVM
used, versus about 200 millisec for the Java hello program.<br>
<br>
See <a href="Hardware-and-software-used-for-Clojure-benchmarks.html">Hardware and software used</a> for details on what was running beneath the JVM in these measurements.&nbsp; The graphs
themselves give information about the JVM and OS used.<br>
<br>
<table style="text-align: left; width: 900px;" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<th>Benchmark name<br>
</th>
<th>Description of what the program should do<br>
</th>
<th>This benchmark stresses ...<br>
</th>
<th>Source code and charts of results<br>
</th>
<th>Extra options to JVM besides "-server -Xmixed -XX:+TieredCompilation -XX:+AggressiveOpts -classpath &lt;path&gt;"<br>
</th>
<th>Additional arguments (after <span style="font-style: italic;">class_name</span>, before "&gt; output_file")<br>
</th>
<th>Input file size (bytes)<br>
</th>
<th>Output file size (bytes)<br>
</th>
</tr>
<tr>
<td>binarytrees<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=binarytrees#about">link</a>]<br>
</td>
<td>Memory allocation and deallocation.&nbsp; Inherently sequential (true?) except if GC can run in parallel.<br>
</td>
<td>[<a href="binarytrees.java-1.java">Java</a>]<br>
[<a href="binarytrees.clojure-rh.clojure">Clojure</a>]<br>
[Results]<br>
</td>
<td>&nbsp;(none)<br>
</td>
<td>20</td>
<td>N/A<br>
</td>
<td>422<br>
</td>
</tr>
<tr>
<td>fannkuchredux<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=fannkuchredux#about">link</a>]<br>
</td>
<td>Generating and counting permutations.&nbsp; Lots of parallelism possible.<br>
</td>
<td>[<a href="fannkuchredux.java">Java</a>]<br>
[<a href="fannkuchredux.clj-12.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx16m<br>
</td>
<td>12<br>
</td>
<td>N/A
</td>
<td>29<br>
</td>
</tr>
<tr>
<td>fasta<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=fasta#about">link</a>]<br>
</td>
<td>Generating
pseudo-random numbers via a particular method using integer arithmetic,
and writing output.&nbsp; Pseudo-random generation portion is
inherently sequential.<br>
</td>
<td>[<a href="fasta.java-4.java">Java</a>]<br>
[<a href="fasta.clj-8.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx16m<br>
</td>
<td>25000000<br>
</td>
<td>N/A
</td>
<td>254,166,745<br>
</td>
</tr>
<tr>
<td>hello<br>
</td>
<td>Print "Hello, world!"<br>
</td>
<td>Startup time<br>
</td>
<td>[<a href="hello.java">Java</a>]<br>
[<a href="hello.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx16m<br>
</td>
<td><br>
</td>
<td>N/A
</td>
<td>14<br>
</td>
</tr>
<tr>
<td>knucleotide<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=knucleotide#about">link</a>]<br>
</td>
<td>Updating hash tables, from small ones to fairly large ones<br>
</td>
<td>[<a href="knucleotide.java-3.java">Java</a>]<br>
[<a href="knucleotide.clojure-7.clojure">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx1024m<br>
</td>
<td>&lt; input_file<br>
</td>
<td>254,166,745</td>
<td>259<br>
</td>
</tr>
<tr>
<td>mandelbrot<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=mandelbrot#about">link</a>]<br>
</td>
<td>Double precision arithmetic, embarassingly parallel<br>
</td>
<td>[<a href="mandelbrot.java-2.java">Java</a>]<br>
[<a href="mandelbrot.clj-9-par-1.3.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx16m<br>
</td>
<td>16000<br>
</td>
<td>N/A
</td>
<td>32,000,015<br>
</td>
</tr>
<tr>
<td>nbody<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=nbody#about">link</a>]<br>
</td>
<td>Double precision arithmetic, inherently sequential<br>
</td>
<td>[<a href="nbody.java-2.java">Java</a>]<br>
[<a href="nbody.clj-14.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx8m<br>
</td>
<td>50000000<br>
</td>
<td>N/A
</td>
<td>26<br>
</td>
</tr>
<tr>
<td>pidigits<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=pidigits#about">link</a>]<br>
</td>
<td>Arbitrary precision integer arithmetic<br>
</td>
<td>[<a href="pidigits.java">Java</a>]<br>
[<a href="pidigits.clojure-2.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx16m<br>
</td>
<td>10000<br>
</td>
<td>N/A
</td>
<td>16,893<br>
</td>
</tr>
<tr>
<td>regexdna<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=regexdna#about">link</a>]</td>
<td>Regex searching and replacement, some parallelism possible<br>
</td>
<td>[<a href="regexdna.java-5.java">Java</a>]<br>
[<a href="regexdna.clj-3.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx616m<br>
</td>
<td>&lt; input_file<br>
</td>
<td>50,833,411<br>
</td>
<td>298<br>
</td>
</tr>
<tr>
<td>revcomp<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=revcomp#about">link</a>]</td>
<td>Reading input, simple transformation of input strings, writing output<br>
</td>
<td>[<a href="revcomp.java-3.java">Java</a>]<br>
[<a href="revcomp.clj-15.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx384m<br>
</td>
<td>&lt; input_file<br>
</td>
<td>254,166,745</td>
<td>254,166,745</td>
</tr>
<tr>
<td>spectralnorm<br>
</td>
<td>[<a href="http://shootout.alioth.debian.org/u64q/performance.php?test=spectralnorm#about">link</a>]</td>
<td>Double precision arithmetic, lots of parallelism possible<br>
</td>
<td>[<a href="spectralnorm.java-2.java">Java</a>]<br>
[<a href="spectralnorm.clojure-6-corrected.clj">Clojure</a>]<br>
[Results]</td>
<td>&nbsp;-Xmx8m<br>
</td>
<td>5500<br>
</td>
<td>N/A
</td>
<td>12<br>
</td>
</tr>
</tbody>
</table>
<br>
<h2><a class="mozTocH2" name="mozTocId86059"></a>Why is -Xmx used?</h2>
The JVM command line option -Xmx&lt;megabytes&gt;m is frequently
used to specify the maximum heap size for these programs, because it
helps reduce the maximum resident memory used by the program.&nbsp;
This is another prominent factor measured and reported on the Computer
Language Benchmarks game web site, in addition to the elapsed time and
total CPU time used.&nbsp; Many Clojure programs that can run fast with
a small maximum heap size often use a much larger amount of resident
memory if permitted, because they allocate memory that quickly becomes
collectible garbage.&nbsp; The JVM will not collect the garbage unless
explicitly triggered by the program, or forced to by a
smaller-than-default maximum heap size.<br>
<h2><a class="mozTocH2" name="mozTocId741195"></a>AOT compilation commands used</h2>
The following commands were used for AOT compilation of all Clojure
programs, where CLJ_SOURCE_FILE is the original source file name,
CLASSNAME is the same as the "Benchmark name" in the table above, and
CLJ_JAR_FILE varied depending upon the Clojure version being measured:<br>
<pre style="margin-left: 40px;">mkdir ./obj</pre>
<pre style="margin-left: 40px;">cp ${CLJ_SOURCE_FILE} ./obj/${CLASSNAME}.clj</pre>
<pre style="margin-left: 40px;">java -Dclojure.compile.path=./obj/clj -classpath ${CLJ_JAR_FILE} clojure.lang.Compile ${CLASSNAME}<br></pre>
<h2><a class="mozTocH2" name="mozTocId677134"></a>Number of Clojure function calls made<br>
</h2>
The table below tells how many calls were made to each Clojure
function, usually with some indication of where most of the execution
time is spent.<br>
<br>
<table style="text-align: left; width: 600px;" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td style="vertical-align: top;">Benchmark / class name<br>
</td>
<td style="vertical-align: top;">Clojure source code<br>
</td>
<td style="vertical-align: top;">Number of times each Clojure function is called<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">binarytrees<br>
</td>
<td style="vertical-align: top;">binarytrees.clojure-rh.clojure<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
bottom-up-tree: 607,475,038<br>
item-check: 607,475,038<br>
<br>
Other calls:<br>
iterate-trees: 9<br>
main: 1<br>
-main: 1</td>
</tr>
<tr>
<td style="vertical-align: top;">fannkuchredux<br>
</td>
<td style="vertical-align: top;">fannkuchredux.clj-12.clj<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
fannkuch-of-permutation: 301,297,917<br>
next-permutation!: 296,930,829<br>
<br>
Other calls:<br>
init-permutations: 1<br>
partial-fannkuch: 12<br>
fannkuch: 1<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">fasta<br>
</td>
<td style="vertical-align: top;">fasta.clj-11.clj<br>
<br>
(TBD: update for fasta.clj-8.clj source file)<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
gen-random-fast: 200,000,000<br>
<br>
Other calls:<br>
find-index: 444,000<br>
make-cumulative: 2<br>
make-lookup-table: 2<br>
cycle-bytes: 1<br>
fasta-repeat: 1<br>
fasta-random: 2<br>
write-line: 3<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">hello<br>
</td>
<td style="vertical-align: top;">hello.clj<br>
</td>
<td style="vertical-align: top;">-main: 1<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">knucleotide<br>
</td>
<td style="vertical-align: top;">knucleotide.clojure-7.clojure<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
tally-dna-subs-with-len: 16<br>
fasta-description-line: 4,166,671<br>
fasta-description-line-beginning: 2,083,337<br>
<br>
Other calls with a small fraction of total run time:<br>
medusa-future-thunk: 16<br>
medusa-pmap: 1<br>
fasta-dna-str-with-desc-beginning: 1<br>
dna-str-to-key: 21<br>
key-to-dna-str: 20<br>
getcnt: 148<br>
tally-total: 2<br>
all-tally-to-str: 2<br>
one-tally-to-str: 5<br>
piece-sizes: 7<br>
break-work-into-pieces: 7<br>
do-one-piece: 16<br>
add-tally-hashmaps!: 2<br>
combine-pieces: 7<br>
run: 1<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">mandelbrot<br>
</td>
<td style="vertical-align: top;">mandelbrot.clj-9-par-1.3.clj<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
dot: 256,000,000<br>
<br>
Other calls:<br>
index-to-val: 32,000<br>
ubyte: 32,000,000<br>
compute-row: 16,000<br>
compute-rows: 1<br>
do-mandelbrot: 1<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">nbody<br>
</td>
<td style="vertical-align: top;">nbody.clj-14.clj<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
v+!: 500,000,000<br>
p-dt!: 250,000,000<br>
v-dt! 500,000,000<br>
advance: 50,000,000<br>
<br>
Other calls:<br>
v!: 1<br>
energy (method of type Body): 10<br>
mk-body: 5<br>
init-state: 1<br>
energy (function): 12<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">pidigits<br>
</td>
<td style="vertical-align: top;">pidigits.clojure-2.clj<br>
</td>
<td style="vertical-align: top;">floor-ev: 86,418<br>
ncomp: 43,209<br>
digit: 43,210<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">regexdna<br>
</td>
<td style="vertical-align: top;">regexdna.clj-3.clj<br>
</td>
<td style="vertical-align: top;">slurp-std-input: 1<br>
count-regex-occurences: 9<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">revcomp<br>
</td>
<td style="vertical-align: top;">revcomp.clj-15.clj<br>
</td>
<td style="vertical-align: top;">Most time is spent in:<br>
reverse-and-complement!: 3<br>
<br>
Other calls:<br>
rba-reverse: 4<br>
find-first-byte-idx: 3<br>
ubyte: 256<br>
make-array-char-mapper: 1<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">spectralnorm<br>
</td>
<td style="vertical-align: top;">spectralnorm.clojure-6-corrected.clj<br>
</td>
<td style="vertical-align: top;">multiply-a-v: 160<br>
multiply-at-v: 160<br>
multiply-at-a-v: 20<br>
run-game: 1<br>
</td>
</tr>
</tbody>
</table>
<br>
</body></html>