Java Benchmark Expressions (JBX)

JBX is a expression written in nix to allow you to compile benchmarks and run analysis on them.

The goal of this repository is to create single instance of a benchmarks, which can be used a fixed point when reasoning about dynamic and static analysis. This would hopefully reduce the time needed to build new tools.

Also the JBX should stand as the platform from which it is possible to compare results of different analyses, both in terms of accuracy, speed, and memory usage.

Usage

Nix

JBX uses nix to manage and compile benchmarks. To get started one would have to download and install nix. According to nix, the easiest way to get started with nix is to run, the following:

$ curl https://nixos.org/nix/install | sh

For instructions about how to use nix refer to their homepage https://nixos.org/nix/.

Vagrant

Alternatively use vagrant. For convinience, if you do not want to install nix on your system, or if you are running a mac or a windows computer, a vagrant file to create a simple nixos has been provided.

WARNING: The vagrant file is only for development use, as running stuff in a vm will impact performance.

First install vagrant (here), then the vagrant plugin vagrant-nixos.

$ vagrant plugin install vagrant-nixos

Then vagrant up and ssh, which should drop you into a shell in the newly installed nixos linux.

$ vagrant up
$ vagrant ssh

The initial setup will take some time.

Structure

The structure of the benchmarks is build in several categories

benchmarks/ : Contains benchmarks that we would like to run code on.

tools/ : Contains expressions about how to compile and download the tools used in the analyses.

analyses/ : Contains all the expressions for running the analyses on benchmarks.

java/ : Contains the different java versions. Java is an important dependency for benchmarks and to fully define java with libraries.

results/ : Contains expressions about applying the analyses onto the benchmarks.

proprietary/ : Does not contain anything but if proprietary tools is used, add the tars here.

-   logicblox:
    To use logicblox3 download `logicblox-3.10.21.tar.gz` and to use
    logicblox4 download `logicblox-4.2.0.tar.gz` and add them to the
    `proprietary/` folder.

-   java: Java 7 and 8 is provided by nixpkg, but to make java 6 and
    java 5 work you have to download `jdk-1_5_0_22-linux-amd64.bin`
    and `jdk-6u45-linux-x64.bin` from oracles homepage and place
    them in the `proprietary/` folder.

Getting started

To use JBX, first clone this repository recursively and checkout a new branch corresponding to your project.

git clone --recursive https://bitbucket.org/ucla-pls/jbx.git
cd jbx
git checkout -b yourProject

NOTE: Downloading the dependencies is very important, so if you did not use the --recursive argument, please run:

git submodule init
git submodule update

For using the jbx tool you have to install python 3.5+ and nix-prefetch-scripts, both can be installed using this command:

nix-env -f nixpkgs/ -i nix-prefetch-scripts python3

Going further: Contributing

It is often not the case the analyses or benchmarks are already there to get the results that you need.

Adding Analyses

Analyses is nix functions that given a benchmark will produce a result. Analyses are ordered by output, e.g. deadlock analyses are stored in analysis.deadlock.

Adding Tools

When adding new tools, first check if they exist in the nixpkgs and before adding them to the project consider adding the tool to the nixpkgs repository. This way a lot more people will use the tool and give feedback. Alternatively adhered to the standards of the nixpkgs repository, and add the nix expression in it's own subfolder.

Adding Benchmarks

If a certain benchmark please feel free to add it to the benchmark/ expression. Benchmarks is ordered after origin, ei. DaCapo benchmarks is under dacapo/. If your benchmark is not associated with any benchmark suite, add it to independent/ expression. Because we are supporting multiple versions of java, we must also use java as a clear dependency when compiling our benchmarks.

A benchmark is jar file (placed relatively in $out/share/java/), which can be executed given a main class and a set of inputs. To create a benchmark you should call mkBenchmark with a minimum of:

• name: a project wide unique identifier.
• mainclass: the main class that execute the program.
• build: which is build source code using a version of java a java library see [#libraries]

Optionally you can add:

• inputs: a list of inputs, the inputs is defined using
  • name: the name of the execution
  • setup: some setup phase before execution. In this phase is the $data and $build variables available.
  • args: a list of arguments. Which can also use the $data variable.
  • stdin: a file containing the stdin of the function.
• data: optional other package which points some data. If not set will this variable point to $build.
• filter: a function that given a Java version 5 through 8, will indicate whether the benchmark run on that version.
• tags: a list of tags which will be used to organise the benchmarks on call.

Any additional arguments is parsed on to eventual analyses.

Good examples of working benchmarks would be dacapo.avrora or dacapo.sunflow, which uses the callBenchmark shorthand which is defined in benchmarks. Also baseline contains minimal code and examples.

Libraries

Many of the benchmarks used will have shared dependencies. They can be defined here.

Comparisons

Included

Tools

Benchmarks

Current results for running petablox on the dacapo benchmarks. All benchmarks are run with the harness.

	Java 5	Java 6	Java 7	Java 8
avrora		✓
batik		✓	✗	✗
eclipse		#11		✗
fop		#13	(✗)	(✗)
h2		✓
jython		#9
luindex		✓
lusearch		✓
pmd		#9
sunflow		✓
tomcat		✓
tradebeans		#13
tradesoap		#13
xalan	✗	✓		✗

Developing

Dacapo

We have tried to keep the version of Dacapo in this repository as close to the original as possible, but we have had to make some changes, the most prominent is the complete removal of the harness from all benchmarks.

Analyses

Analyses are the wrappers around the tools to make them callable from nix. An analysis takes only one argument and that is the benchmark. The analysis SHOULD as such be self contained, and all analyses is considered deterministic, If the analyses is not, make sure to run it enough times, to get a representative result. It is allowed to create functions that create analyses, by parsing setup arguments.

All analyses MUST NOT have side effect, if so find or run the program in a rooted environment.

Useful commands

Command	Description
nix-env -f expr -iA tools.?	Install the tool so that you can use it everywhere

To Do

• Add Dacabo to the benchmarks.

• Create a tutorial or tool chain for how to run analyses with side effect.