Skip to content
An Extensible Java Bug Benchmark for Automatic Program Repair Studies
Branch: master
Clone or download
Latest commit 0c8be2c Feb 27, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.

The Bears-Benchmark

The Bears-Benchmark, or just Bears, is a benchmark of bugs for automatic program repair studies in Java. The bugs are collected from open-source projects hosted on GitHub through a process that scans pairs of builds from Travis Continuous Integration and reproduces bugs (by test failure) and their patches (passing test suite).

If you use Bears, please cite our paper:

  author = {Fernanda Madeiral and Simon Urli and Marcelo Maia and Martin Monperrus},
  title = {{Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies}},
  booktitle = {Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER '19)},
  year = {2019},
  url = {}

Bug Repository Design

Each bug is stored in a branch. The name of each branch follows the pattern <GitHub project slug>-<buggy Travis build id>-<patched Travis build id>. The files and data on a bug are organized in commits in the branch of the bug as follows:

  • Commit #1 contains the version of the program with the bug
  • Commit #2 contains the changes in the tests
  • Commit #3 contains the version of the program with the human-written patch
  • Commit #4 contains the metadata file bears.json, which is a gathering of information collected during the bug reproduction process. It contains information about the bug (e.g. test failure names), the patch (e.g. patch size), and the bug reproduction process (e.g. duration).

How to use

In the folder scripts you can find scripts to check out and compile bugs.

How to contribute

Any contribution is very welcome! The types of contribution are, but not limited to:

  • The proposal of bugs to be added in Bears (specific instructions comming soon)
  • To flag possibly incorrect branches added in the Bears
  • To participate in the manual validation of bugs/branches, i.e. review open pull requests


Feel free to create issues in this repository with questions or to drop an email to

You can’t perform that action at this time.