Note. The Train Benchmark has a fork for the 2015 Transformation Tool Contest, primarily targeting EMF tools. This repository contains the original Train Benchmark which also supports RDF, SQL and property graph databases.
Warning. The Train Benchmark is designed to run in a server environment. Some implementations may shut down or delete existing databases, so only run it on your developer workstation if you understand the consequences.
For theoretical and implementation details, check out the following documents:
The generator projects are responsible for generating instance models. Currently, the following formats are supported:
- EMF
- Property graph
- RDF
- SQL
The benchmark projects are responsible for running the benchmarks.
- EMF
- Drools 5 & 6
- EMF API
- EMF-IncQuery
- Eclipse OCL
- Property graph
- Neo4j
- OrientDB
- RDF
- Blazegraph
- Jena
- Sesame
- Virtuoso
- SQL
- MySQL
The framework provides a set of scripts for building the projects, generating the instance models and running the benchmark.
The benchmark requires a 64-bit operating system. We recommend Ubuntu-based Linux systems.
Provided that you start with a fresh Ubuntu server installation, you may use the provided install scripts:
scripts/init-jdk.sh && \
scripts/init-maven.sh && \
scripts/init-python.sh && \
scripts/dep-mysql.sh && \
scripts/dep-neo4j.sh && \
scripts/dep-virtuoso.sh && \
scripts/dep-mondo-sam.sh
Alternatively, install the following software:
- Oracle JDK 8
- Maven 3
- Python 3
- MySQL
- Neo4j
- Virtuoso
- MONDO-SAM 0.1
Initialize the configuration file by running:
scripts/init-config.sh
This creates config/config.yml
which defines the configuration for the benchmark. The documentation is provided as comments in the file.
The scripts
directory contains the run.py
script which is used for the following purposes:
scripts/run.py -b
-- build the projectsscripts/run.py -b -s
-- build the projects without testingscripts/run.py -g
-- generates the instance modelsscripts/run.py -m
-- runs the benchmarkscripts/run.py -h
-- displays the help
The projects are developed and tested with Eclipse Mars.
To import and develop the Train Benchmark, you need the m2e Eclipse plugin, included in Eclipse for Java developers. If you use another distribution (e.g. Eclipse Modeling), you can install it from the Mars update site or the m2e update site (http://download.eclipse.org/technology/m2e/releases).
To avoid confusion between the different implementations, we decided to use the Smurf Naming convention (see #21). For example, the classes in the Java implementation are named JavaBenchmarkCase
, JavaPosLength
, JavaPosLengthMatch
, JavaPosLengthTransformation
, while the classes in the EMF-IncQuery implementation are named EMFIncQueryBenchmarkCase
, EMFIncQueryPosLength
, etc. We found that relying on the package names to differentiate class names is error-prone and should be avoided.
Follow the instructions here.
It is possible to convert the measurement results from JSON to CSV with the following script:
scripts/convert-results.sh
In order to use the interactive interface in MONDO-SAM, install additional R packages as described here, then run the following:
scripts/interactive.py
Adjust the config/reporting.json
file and run the scripts/report.sh
script. The possible configuration values are listed in the MONDO-SAM wiki.
The Train Benchmark provides two sorts of instance models:
- Minimal models, used only for testing
- Scalable models, used both for testing and benchmarking
The minimal models contain only a few (8-10) model elements to provide simple models for development and testing.
The scalable models are generated for each scenario in sizes denoted by the powers of two, e.g. railway-repair-1
, railway-repair-2
, railway-repair-4
, etc.