MUBench (pronounced "Moo Bench") is an automated benchmark for API-misuse detectors, based on the MUBench benchmarking dataset. If you encounter any problems using MUBench, please report them to us. If you have any questions, please contact Sven Amann.
The people (in alphabetical order by their last name) who actively contributed to the concept, the dataset, or the codebase of MUBench since the project's inception in October 2015:
- Sven Amann (Project Lead, Since 10/2015)
- Mattis Kämmerer (Developer, 03/2016 - 03/2018)
- Sarah Nadi (Dataset Reviewer, 10/2015 - 05/2016)
- Hoan Anh Nguyen (Dataset Reviewer, 10/2015 - 05/2016)
- Tien N. Nguyen (Dataset Reviewer, 10/2015 - 05/2016)
- Jonas Schlitzer (Developer, 10/2016 - 07/2018)
The MUBench project was developed at the Software Technology Group of Technische Universität Darmstadt from October 2015 to September 2018. From October 2018, it is being maintained by Sven Amann as an independent Open Source project.
- 'Exposing Library API Misuses via Mutation Analysis' (ICSE '19)
- 'Investigating Next Steps in Static API-Misuse Detection' (MSR '19)
- 'A Systematic Evaluation of Static API-Misuse Detectors' (TSE '18)
- 'MUBench: A Benchmark for API-Misuse Detectors' (MSR '16 Data Showcase)
We provide instructions to reproduce the MUBench experiments presented in the above publications.
With MUBench, you may run different API-misuse detectors in a number of experiments to determine their precision and recall. To this end, MUBench provides a curated dataset of real-world projects and known misuses. In each experiment run, the respective detector emits findings which you need to review manually. To this end, MUBench publishes (a subset of) the findings to a review website. After you completed your reviews, the site automatically computes experiment statistics.
Afterwards, you can start the MUBench Interactive Shell using
$> docker run -it -v mubench-checkouts:/mubench/checkouts -v mubench-findings:/mubench/findings --rm -p 8080:80 svamann/mubench:stable
This docker command starts in interactive shell (
-it) based on our Docker image
svamann/mubenchin the latest
stableversion. It persists experiment data on Docker volumes named
-v), while disposing of all other state on exit (
--rm). And it forwards port
80from within the shell to port
8080of your host system, to allow running a standalone review site.
(Optional) You may want to create a script with the above command, which allows you to conveniently open a MUBench Interactive Shell by typing
./mubenchor running individual commands by typing
(Optional) You may want to setup a review site to collaboratively review detector findings and publish your results.
Hint: We recommend to use the latest stable version
svamann/mubench:stable of our Docker image.
However, you may also chose to use the latest development version
svamann/mubench:latest, which is continuously deployed from the
master branch of this repository.
For all usage examples in this documentation, we assume that you opened a MUBench Interactive Shell. Alternatively, you may execute individual commands by passing them as arguments to the docker command for running MUBench.
- Run experiments.
- Publish misuse metadata to a review site.
- Publish detector findings to a review site.
- Debug a detector (runner).
We want MUBench to grow, so please be welcome to
- Add your own projects or misuses to the benchmarking dataset.
- Add your own detector to the benchmark.
- Contribute to the benchmarking platform
All software provided in this repository is subject to the CRAPL license.
The detectors included in MuBench are subject to the licensing of their respective creators. See the information in the detectors' folders.
The projects referenced in the MuBench dataset are subject to their respective licenses.
The project artwork is subject to the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).