Skip to content
BenchExec: A Framework for Reliable Benchmarking and Resource Measurement
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchexec Fix bug in HTML table: substring search instead of regexp search May 21, 2019
bin Add containerexec: a utility for executing commands in a container Apr 22, 2016
contrib Add exception handling Feb 1, 2019
debian Use LXCFS if available for better container isolation May 17, 2019
doc PyPI now supports Markdown, we no longer nead to convert it May 21, 2019
test Use LXCFS if available for better container isolation May 17, 2019
.coveragerc Attempt to fix Travis failure because of coverage error Dec 28, 2016
.gitattributes Fix gitattributes such that GitHub ignores the test files for its lan… Mar 9, 2017
.gitignore add .idea (PyCharm IDE) to .gitignore Nov 25, 2018
.gitlab-ci.yml Use LXCFS if available for better container isolation May 17, 2019
.project Initialize Eclipse project. Feb 13, 2015
.pydevproject Initialize Eclipse project. Feb 13, 2015
.travis-setup.sh fix test for blkio measurements and enable them on Travis Jan 5, 2017
.travis.yml Remove test coverage from Travis CI Feb 14, 2019
CHANGELOG.md Changelog for next version 1.18 Feb 11, 2019
LICENSE Initial commit Feb 13, 2015
MANIFEST.in Move table-generator tests around to not other hide tests among the Jan 12, 2017
README.md Revert "Add link to survey to our readme." May 17, 2019
release.sh Make our wheel builds reproducible May 21, 2019
setup.py PyPI now supports Markdown, we no longer nead to convert it May 21, 2019

README.md

BenchExec

A Framework for Reliable Benchmarking and Resource Measurement

Build Status Code Quality Test Coverage Apache 2.0 License PyPI version DOI

News:

BenchExec provides three major features:

  • execution of arbitrary commands with precise and reliable measurement and limitation of resource usage (e.g., CPU time and memory), and isolation against other running processes
  • an easy way to define benchmarks with specific tool configurations and resource limits, and automatically executing them on large sets of input files
  • generation of interactive tables and plots for the results

Contrary to other benchmarking frameworks, it is able to reliably measure and limit resource usage of the benchmarked tool even if it spawns subprocesses. In order to achieve this, it uses the cgroups feature of the Linux kernel to correctly handle groups of processes. For proper isolation of the benchmarks, it uses (if available) Linux user namespaces and an overlay filesystem to create a container that restricts interference of the executed tool with the benchmarking host. BenchExec is intended for benchmarking non-interactive tools on Linux systems. It measures CPU time, wall time, and memory usage of a tool, and allows to specify limits for these resources. It also allows to limit the CPU cores and (on NUMA systems) memory regions, and the container mode allows to restrict filesystem and network access. In addition to measuring resource usage, BenchExec can verify that the result of the tool was as expected, and extract further statistical data from the output. Results from multiple runs can be combined into CSV and interactive HTML tables, of which the latter provide scatter and quantile plots (have a look at our demo table).

BenchExec works only on Linux and needs a one-time setup of cgroups by the machine's administrator. The actual benchmarking can be done by any user and does not need root access.

BenchExec was originally developed for use with the software verification framework CPAchecker and is now developed as an independent project at the Software Systems Lab of the Ludwig-Maximilians-Universität München (LMU Munich).

Links

Authors

Maintainer: Philipp Wendler

Contributors:

Users of BenchExec

BenchExec was successfully used for benchmarking in all instances of the international competitions on Software Verification and Software Testing with a wide variety of benchmarked tools and hundreds of thousands benchmark runs. It is integrated into the cluster-based logic-solving service StarExec (GitHub).

The developers of the following tools use BenchExec:

If you would like to be listed here, contact us.

You can’t perform that action at this time.