This repository contains benchmarks for our composed Python/PHP VM as well as other VMs.
If you are using the VirtualBox image, you can skip this stage, as the VMs come pre-built.
To build yourself, first ensure you have the following installed on a Linux/amd64 system:
- GCC (and gcc-multilib).
- Dependencies of HHVM, PyPy, and HippyVM.
$ sh build.sh
Running the Benchmarks
First inspect the generated
config.py. Here you can adjust (for example)
the number of iterations at each Kalibera level and the benchmark parameters.
In each VM dict the parameter
n_iterationssets the number of in-process iterations on a per-VM basis..
N_EXECUTIONSparameter sets how many process executions to repeat with.
The defaults you see in the config file are the parameters we used.
To run the benchmarks:
$ make bench
Results are written to
The source code for the benchmarks is in the 'benchmarks/' directory.
On our circa 2014 benchmark machines, running the benchmarks took several days.
The tables from the paper can generated by running
To make tables you need a
config_results.json file to be present in this
directory. To generate this file, run the experiment as shown above.
Alternatively, if you are using the VirtualBox image, then the data from our
run can be found in ~/raw_results. You can copy the
here and generate tables from this.
Note that the bootstrapping method we use to compute confidence intervals  is non-deterministic, so even if you use our data, expect small differences with the tables you see in the paper.
Generating the tables requires a working TeX Live setup.
The two case studies from the paper are also included.
If you are using the VirtualBox image, then the case studies come ready to run. Please see the post-login message for more information.
To set up the case studies manually, run:
$ cd casestudies & make
PyHyP needs to have the environment variable
PYPY_PREFIX set to the
pypy-hippy-bridge folder. If you don't want to set this variable globally, you
can just patch the
hippyvm/hippy-c-cgi executable by adding the appropriate
#!/bin/sh export PYPY_PREFIX=PATH_TO_EXPERIMENTS/pypy-hippy-bridge/ HIPPY=`dirname $0`/hippy-c exec $HIPPY --cgi "$@"
To run the case studies you need to setup a web-server, e.g. Apache, Lighttpd, nginx. You'll then have to replace the PHP interpreter with PyHyp. Here's an example configuration for lighttpd:
# /etc/lighttpd/conf.d/cgi.conf cgi.assign = ( ".pl" => "/usr/bin/perl", ".cgi" => "/usr/bin/perl", ".rb" => "/usr/bin/ruby", ".erb" => "/usr/bin/eruby", ".php" => "PATH_TO_EXPERIMENTS/hippyvm/hippy-c-cgi", ".py" => "/usr/bin/python2.7" )
configure to configure Squirrelmail. At the very least you will need to
change the following things:
- Enter your IMAP Server details in
2. Server-Settings -> Update IMAP Settings.
- Change the
4. General Options -> Data Directoryto
../data. Then change the ownership of
squirrelmail-4.22.1/data/to be accessible by your webserver user, e.g.
chown -R http:http data/.
Finally, point the web server to the SquirrelMail folder
casestudies/squirrelmail-webmail-1.4.22/). Here's a minimal configuration
file for lighttpd:
Activating the plugin
configure once more:
cd squirrelmail-webmail-1.4.22/ ./configure
8. Plugins, then select
Activate HTML emails
To be able to see rendered formulae within Squirrelmail we need to activate HTML
viewing of emails which is deactived by default. To do that, go to the local
Squirrelmail website, then navigate to
Options->Display Preferences and check
Show HTML Version by default and
Sympy is missing
To be able to use Sympy we need to install it into PyHyp.
The easiest way to do this is to download the current Sympy version from
sympy folder from the archive into
This example can be run just using the PyHyp executable. Simply run:
PYPY_PREFIX=../work/pyhyp/pypy-hippy-bridge ../work/pyhyp/hippyvm/pyhyp cffi.php
 Quantifying Performance Changes with Effect Size Confidence Intervals, Tomas Kalibera and Richard Jones, 2012.