Getting Started with Adapton's OOPSLA'15 Artifact
Install a Program to run the virtual machine
- We used VirtualBox with default settings
- Go to https://www.virtualbox.org/wiki/Downloads
- Select the appropriate option for your platform
- Install the program
Import Adapton's VM image file:
From Menu: File->Import Appliance...
Run the VM and log on
Double-click the icon in the left pane
A window will open with Arch Linux (command-line only; there is no internal windowing system).
Hit enter or wait a few seconds to advance from the boot screen
Enter the user/password: guest/guest
ic login: guest
Password: guest (text hidden)
Run our test script:
[guest@ic ~]$ ./recreate-results.sh
- This will take 5-10 minutes
Inspect the generated tables:
[guest@ic ~]$ less table1-results.txt
[guest@ic ~]$ less table2-results.txt
The format of these two table files matches that of the tables in the paper (tables 1 and 2).
However, to generate these tables quickly, the
script performs fewer samples and runs our benchmarks on smaller
inputs than in the paper's tables. For this reason, the results from
this script will not generally match the tables in the paper. In
particular, our speedups generally increase with increased input
sizes. (Instructions to create the full experiments are below).
We also welcome you to perform interactive tests with more demanding parameters, described further below.
All our results reported in the paper were from a 2.26 GHz Intel Mac
Pro with 16 GB of RAM running Mac OS X 10.6.8. Our original data can
be found in
oopals15_data_log/. We measure the time
taken to perform calculations both processor intensive and memory
intensive. Results may vary greatly depending on the system used. The
trends should not vary as much, so we encourage reviewers to consider
speedups rather than time taken.
(Optional) Recreate the full experiments we reported in our paper.
Run the script above with the
Unlike the short version described above, this invocation of the script (using
full) will take most of a day to run, and will require different VM settings. In particular, you'll need to change settings on the VM to allow 8 GB of memory (the default is 4 GB).
Our original raw data is in
Let us know if you have any trouble!
Step by step guide to Adapton's OOPSLA'15 Artifact
The Getting Started guide above automatically collects experimental data and re-creates tables from the paper. This section explains how to run individual experiments manually, and explores the other content of the VM image.
List-based (and Tree-based) Tests
The test script we ran above is
script/test-oopsla15.sh. This script
begins with a list of options, which can be edited manually.
The following sequence will generate results:
[guest@ic ~]$ cd adapton.ocaml/
[guest@ic adapton.ocaml]$ ./experiment.native --experiment Rope_mergesort_lazyrecalc --0 --r --num-changes 1 --n 50000
[guest@ic adapton.ocaml]$ ./experiment.native --experiment Rope_mergesort_name --0 --r --num-changes 1 --n 50000
--help for help:
[guest@ic adapton.ocaml]$ ./experiment.native --help
For more details, see our supplementary material.
The tests above consist of larger input sizes than those reported in the paper. Each invocation only performs a single sample. We see that Non-Adapton ('lazyrecalc') takes much less time to generate the initial list, but far more time to make an incremental change than Adapton ('name'). The output on screen is sufficient to see a difference; the full measurements can be found in an output file:
[guest@ic adapton.ocaml]$ cd out/
[guest@ic adapton.ocaml]$ tail experiments.csv
Here each test has a line for initial construction, one for each change made, and one for final heap size. The time is the second floating point number (the first one with some decimal digits).
The imp test script that generates data for table2 is
test.py. You can edit the common parameters at the top:
[guest@ic ~]$ cd ~/incremental-computation
[guest@ic incremental-computation]$ nano test.py
Nano has common commands listed at the bottom. Press control-x to exit and y [enter] to save your work.
The parameters at the top are set to run simpler tests that finish
intl_input must be powers of 2. The second
group of parameters are the ones used in the paper submission. This
script is run as part of the test set above. You can call it as above,
or add a parameter to run the longer set of tests
reproduce-results.sh full. This will take many hours, though.
Tests can also be run individually. We don't suggest doing this unless you want to deal with the details of how our tests work, but the following sequence will produce results:
[guest@ic incremental-computations]$ ./imptests.native --artlib fromscratch --test fact3 --record --fact 10000
[guest@ic incremental-computations]$ ./imptests.native --artlib structural --test fact3 --record --fact 10000
[guest@ic incremental-computations]$ ./imptests.native --artlib nominal --test fact3 --record --fact 10000
[guest@ic incremental-computations]$ cd results
[guest@ic incremental-computations]$ tail fact-begin-swap-assign.csv
What you see is the raw output file. The last three lines are the ones we just generated. They are more demanding but less thorough then the paper's results. The first floating point number in each line is the initial program run time. The second floating point number is the incremental run time. As you can see, Adapton ('structural', 'nominal') takes slightly longer for the initial program than Non-Adapton ('fromscratch'). The benefit is in the incremental change runtime, where Adapton performs far better.
[guest@ic ~]$ ls
Our main Adapton git repo has been cloned to
[guest@ic ~]$ cd adapton.ocaml/
It is also available publicly: https://github.com/plum-umd/adapton.ocaml
script/-- sample scripts for correctness and performance testing.
Source/-- our implementation.
Sample/-- example stub code for new projects.
out/-- results from scripts, after they run.
The following is a clone of our development repository for the IMP interpreter:
[guest@ic adapton.ocaml]$ cd ~/incremental-computation
imptests.native-- the testing executable (compiled OCaml code).
test.py-- a helper testing script.
results/-- results directory, with raw output measurements.
imp/-- the OCaml code being tested.
incremental-computation directories each have targets for
make test-- rebuilds our test program
make test-db-- rebuilds our test program
experiments.byte(bytecode with debugging symbols)
make opam-pin-- builds adapton library and use
opamto install it locally
make opam-reload-- use after
opam-pinhas been run once
make-- rebuilds our test programs