This is the readme file of the ARCH2018 Repeatibility Evaluation (RE) package. Instructions are provided for installing and running the benchmarks.
ARCH18_RE.jl is just a "wrapper" module for running the benchmarks of this RE.
The actual algorithms belong to the package Reachability.jl.
This year we have considered purely continuous models only. These are:
- International Space Station (ISS) alias
- Building model alias
Overview. To install and run the benchmarks, basically these three steps are needed:
- Download and install Julia using the official download links.
Reachability.jland its dependencies. For that purpose we provide the script
- Run the benchmarks and obtain results. For that purpose we provide the script
run.sh. The results are saved in the file
Below we explain these steps in some detail. A section on known issues is given in the end. If you still have problems or questions, do contact us in our gitter channel or via email.
JuliaReach is a software framework for reachability computations of dynamical systems, available at the JuliaReach github project website. It is written in Julia, a modern high-level language for scientific computing. Currently JuliaReach can handle continuous affine systems. The reachability algorithm uses a block decomposition technique presented in DBLP:conf/hybrid/BogomolovFFVPS18. Here we partition the state space, project the initial states to subspaces, and propagate these low-dimensional sets in time. This allows us to perform otherwise expensive set operations in low dimensions. Furthermore, if the output does not depend on all dimensions, we can effectively skip the reach set computation for the respective dimensions. In the evaluation we used two-dimensional blocks, for which our implementation supports epsilon-close approximation; for box approximation we can handle arbitrary partitions. For the set computations we use the LazySets library, which is also part of the JuliaReach framework. LazySets exploits the principle of lazy (on-demand) evaluation and uses support functions to represent lazy sets. JuliaReach also comes with SX, a parser for SX (SpaceEx format) model files. For next year we plan to add support for hybrid dynamics, which will require a careful balance between low- and high-dimensional computations and adaptive choice of the partition.
This package requires Julia v0.6.x. Refer to the official documentation on how to install and run Julia in your system.
Once you have installed Julia, you should be able to open it in a terminal (shell)
with the command
julia, and see the welcome message similar to
$ julia _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_) | Documentation: https://docs.julialang.org _ _ _| |_ __ _ | Type "?help" for help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.6.1 (2017-10-24 22:15 UTC) _/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release |__/ | x86_64-apple-darwin14.5.0 julia>
To install the reachability API we provide the shell script
in this folder. Execute the script
install.sh with the command:
Note. The script
install.sh evaluates the following commands in your Julia installation:
julia> Pkg.clone("https://github.com/JuliaReach/Reachability.git") julia> Pkg.clone("https://github.com/JuliaReach/ARCH2018_RE.git")
Reachability.jl has several dependencies and the other packages also
have their dependencies, this whole process will take between 5-20 minutes in a
fresh Julia installation.
Running the benchmarks
To run and get the benchmark results in the file
results.md, execute the
run.julia with the command:
$ julia --color=yes run.jl
results.md file is saved in your current working directory.
Note. To obtain statistically meaningful performance measures, each reachability computation is performed several times. The script can take several minutes.
In the results tables, the reported time is the minimum time in seconds for each
safety property check. Other statistics such as mean and max values can be found
by looking at the entries of the
For more advanced testing settings we refer to the official BenchmarkTools documentation.
To be able to run the shell script, you may have to give the appropriate permissions. In Linux and MacOS systems this is done with the command:
$ chmod 744 install.sh
If running one scripts give failures, try running it a second time.
See also the installation instructions of the LazySets.jl library.
If you still have trouble, don't hesitate to contact us in our gitter channel.