Stabilizer: Statistically Rigorous Performance Evaluation
Copyright (C) 2013 University of Massachusetts Amherst
Stabilizer is a compiler transformation and runtime library for dynamic memory layout randomization. Programs built with Stabilizer run with randomly-placed functions, stack frames, and heap objects. Functions and stack frames are moved repeatedly during execution. A random memory layout eliminates the effect of layout on performance, and repeated randomization leads to normally-distributed execution times. This makes it straightforward to use standard statistical tests for performance evaluation.
A more detailed description of Stabilizer is available in the Paper, which will appear at ASPLOS 2013 in March.
Stabilizer requires LLVM 3.1. Stabilizer runs on OSX and Linux, and supports x86, x86_64, and PowerPC.
Stabilizer requires LLVM 3.1. Follow the directions here to build LLVM 3.1 and the Clang front-end. Stabilizer's build system assumes LLVM include files will be accessible through your default include path.
By default, Stabilizer will use GCC and the Dragonegg plugin to produce LLVM IR. Fortran programs can only be built with the GCC front end. Stabilizer is tested against GCC version 4.6.2.
Stabilizer's compiler driver
szc is written in Python. It uses the
argparse module, so a relatively modern version of Python (>=2.7) is required.
$ git clone git://github.com/ccurtsinger/stabilizer.git stabilizer $ make
By default, Stabilizer is build with debug output enabled. Run
make clean release to build the release version with asserts and debug output
Stabilizer includes the
szc compiler driver, which builds programs using the
Stabilizer compiler transformations.
szc passes on common GCC flags, and is
compatible with C, C++ and Fortran inputs.
To compile a program in
foo.c with Stabilizer, run:
$ szc -Rcode -Rstack -Rheap foo.c -o foo
-R flags enable randomizations, and may be used in any combination.
Stabilizer uses GCC with the Dragonegg plugin as its default front-end. To
use clang, pass
The resulting executable is linked against with
on OSX). Place this library somewhere in your system's dynamic library search
path or (preferably) add the Stabilizer base directory to your
DYLD_LIBRARY_PATH environment variable.
szclo.cfg config files can be installed in a SPEC CPU2006
config directory to build and run benchmarks with Stabilizer. The szchi config
-O2 for base and
-O3 for peak tuning, and szclo uses
process.py scripts were used to drive experiments and
collect results. The run script accepts optimization levels, benchmarks to
enable (or disable with a "-" prefix), a number of runs, and build
configurations in any order. For example:
$ ./run.py 10 bzip2 code code.stack code.heap.stack
This will run the
bzip2 benchmark 10 times in each of three randomization
runspec tool must be in your path, so
cd to your SPEC
sourceh shrc first.
$ ./run.py 10 -astar code link O2 O3
This will run every benchmark except
astar 10 times with link randomization
-O3 optimization levels.
Be warned: there is no easy way to distinguish
O0 results after the
fact: both are marked as "base" tuning. Keep these results in separate
The process script reads
.rsf files from SPEC and provides some summary
statistics, or collects results in an easy-to-process format.
$ ./process.py $SPEC/result/*.rsf
This will print average runtimes for each benchmark in each configuration and tuning level for the runs in your SPEC results directory.
-trim flag to remove the highest and lowest runtimes before computing
-norm flag tests the results for normality using the Shapiro-Wilk test.
-all flag dumps all results to console, suitable for pasting into a
spreadsheet or CSV file.
Stabilizer is distributed under the GNU GPLv2 license. Contact firstname.lastname@example.org if you are interested in licensing Stabilizer for commercial use.