Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve documentation #156

Merged
merged 3 commits into from
May 20, 2018
Merged

Improve documentation #156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ebfa32b
Improve documentation
mernst May 9, 2018
e4ec5a2
More reorganization of documentation
mernst May 9, 2018
b1f4fd5
More documentation improvements
mernst May 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 85 additions & 66 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
Defects4J -- version 1.2.0 [![Build Status](https://travis-ci.org/rjust/defects4j.svg?branch=master)](https://travis-ci.org/rjust/defects4j)
----------------
================
Defects4J is a collection of reproducible bugs and a supporting infrastructure
with the goal of advancing software engineering research.

Contents of Defects4J
================

The projects
---------------
Defects4J contains 395 bugs from the following open-source projects:
Expand All @@ -22,8 +25,9 @@ Each bug has the following properties:

- Issue filed in the corresponding issue tracker, and issue tracker identifier
mentioned in the fixing commit message.
- Fixed in a single commit -- the Defects4J maintainers manually pruned out
irrelevant changes (e.g., refactorings or feature additions).
- Fixed in a single commit
- Minimized: the Defects4J maintainers manually pruned out
irrelevant changes in the commit (e.g., refactorings or feature additions).
- Fixed by modifying the source code (as opposed to configuration files,
documentation, or test files).
- A triggering test exists that failed before the fix and passes after the fix
Expand All @@ -32,11 +36,14 @@ Each bug has the following properties:
The (b)uggy and (f)ixed program revisions are labelled with `<id>b` and
`<id>f`, respectively (`<id>` is an integer).

Setting up Defects4J
================

Requirements
----------------
- Java 1.7
- Perl >= 5.0.10
- Perl modules: run `cpan Bundle::CSV DBI` or `sudo cpan Bundle::CSV DBI`
- Perl modules: run `cpan Bundle::CSV DBD::CSV DBI` or `sudo cpan Bundle::CSV DBD::CSV DBI`
- Git >= 1.9
- SVN >= 1.8

Expand All @@ -50,9 +57,9 @@ Defects4J generates and executes tests in the timezone `America/Los_Angeles`.
If you are using the bugs outside of the Defects4J framework, export the `TZ`
environment variable accordingly.

Getting started
Steps to set up Defects4J
----------------
#### Setting up Defects4J

1. Clone Defects4J:
- `git clone https://github.com/rjust/defects4j`

Expand All @@ -63,57 +70,45 @@ Getting started
3. Add Defects4J's executables to your PATH:
- `export PATH=$PATH:"path2defects4j"/framework/bin`

#### Using Defects4J
4. Check installation and get information for a specific project (commons lang):
4. Check installation:
- `defects4j info -p Lang`

Using Defects4J
================

#### Example commands
1. Get information for a specific project (commons lang):
- `defects4j info -p Lang`

5. Get information for a specific bug (commons lang, bug 1):
2. Get information for a specific bug (commons lang, bug 1):
- `defects4j info -p Lang -b 1`

6. Checkout a buggy source code version (commons lang, bug 1, buggy version):
3. Checkout a buggy source code version (commons lang, bug 1, buggy version):
- `defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy`

7. Change to the working directory, compile sources and tests, and run tests:
4. Change to the working directory, compile sources and tests, and run tests:
- `cd /tmp/lang_1_buggy`
- `defects4j compile`
- `defects4j test`

8. More examples of how to use the framework are available in `framework/test`
5. The scripts in [`framework/test/`](tree/master/framework/test/)
are examples of how to use Defects4J, which you might find useful
as inspiration when you are writing your own scripts that use Defects4J.

Publications
------------------
* "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs"
René Just, Darioush Jalali, and Michael D. Ernst,
ISSTA 2014 [[download]][issta14].

* "Are Mutants a Valid Substitute for Real Faults in Software Testing?"
René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser,
FSE 2014 [[download]][fse14].

[issta14]: https://people.cs.umass.edu/~rjust/publ/defects4j_issta_2014.pdf
[fse14]: https://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf

Documentation
--------------------
Detailed documentation for any script or module is available as
[html documentation][htmldocs].

[htmldocs]: http://people.cs.umass.edu/~rjust/defects4j/html_doc/index.html

Command-line interface
Command-line interface: defects4j command
-----------------------
Use `framework/bin/defects4j` to execute any of the following commands:
Use [`framework/bin/defects4j`](http://people.cs.umass.edu/~rjust/defects4j/html_doc/defects4j.html) to execute any of the following commands:

| Command | Description |
|----------------|---------------------------------------------------------------------------------------------------|
| info | View configuration of a specific project or summary of a specific bug |
| checkout | Checkout a buggy or a fixed project version |
| compile | Compile sources and developer-written tests of a buggy or a fixed project version |
| test | Run a single test method or a test suite on a buggy or a fixed project version |
| mutation | Run mutation analysis on a buggy or a fixed project version |
| coverage | Run code coverage analysis on a buggy or a fixed project version |
| monitor.test | Monitor the class loader during the execution of a single test or a test suite |
| export | Export version-specific properties such as classpaths, directories, or lists of tests |
| [info](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-info.html) | View configuration of a specific project or summary of a specific bug |
| [checkout](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-checkout.html) | Checkout a buggy or a fixed project version |
| [compile](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-compile.html) | Compile sources and developer-written tests of a buggy or a fixed project version |
| [test](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-test.html) | Run a single test method or a test suite on a buggy or a fixed project version |
| [mutation](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-mutation.html) | Run mutation analysis on a buggy or a fixed project version |
| [coverage](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-coverage.html) | Run code coverage analysis on a buggy or a fixed project version |
| [monitor.test](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-monitor.test.html) | Monitor the class loader during the execution of a single test or a test suite |
| [export](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-export.html) | Export version-specific properties such as classpaths, directories, or lists of tests |


Export version-specific properties
Expand All @@ -140,17 +135,54 @@ provides the following scripts:

| Script | Description |
|-------------------|-----------------------------------------------------------------|
| run_bug_detection | ^Determine the real fault detection rate |
| run_mutation | ^Determine the mutation score |
| run_coverage | ^Determine code coverage ratios (statement and branch coverage) |
| run_evosuite | Generate test suites using EvoSuite |
| run_randoop | Generate test suites using Randoop |
^Note that this script requires Perl DBI.

Directory structure
| [defects4j](http://people.cs.umass.edu/~rjust/defects4j/html_doc/defects4j.html) | Main script, described above |
| [run_bug_detection](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_bug_detection.html) | Determine the real fault detection rate |
| [run_mutation](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_mutation.html) | Determine the mutation score |
| [run_coverage](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_coverage.html) | Determine code coverage ratios (statement and branch coverage) |
| [run_evosuite](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_evosuite.html) | Generate test suites using EvoSuite |
| [run_randoop](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_randoop.html) | Generate test suites using Randoop |

Additional resources
================

Scripts built on Defects4J
--------------------

#### Fault localization (FL)
- [Scripts and annotations for evaluating FL techniques][FL-eval]

#### Automated program repair (APR)
- [Scripts and annotations for evaluating APR techniques][APR-eval]
- [Patches generated with the Nopol, jGenProg, and jKali APR systems][APR-patches-spirals]

[fl-eval]: https://bitbucket.org/rjust/fault-localization-data
[APR-eval]: https://github.com/LASER-UMASS/AutomatedRepairApplicabilityData
[APR-patches-spirals]: https://github.com/Spirals-Team/defects4j-repair

Publications
------------------
* "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs"
René Just, Darioush Jalali, and Michael D. Ernst,
ISSTA 2014 [[download]][issta14].

* "Are Mutants a Valid Substitute for Real Faults in Software Testing?"
René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser,
FSE 2014 [[download]][fse14].

[issta14]: https://people.cs.umass.edu/~rjust/publ/defects4j_issta_2014.pdf
[fse14]: https://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf

[More publications](https://scholar.google.com/scholar?q=defects4j)

Implementation details
----------------------
This is the top-level directory of Defects4J.
The directory structure is as follows:

Documentation for any script or module is available as
[html documentation][htmldocs].

[htmldocs]: http://people.cs.umass.edu/~rjust/defects4j/html_doc/index.html

The directory structure of Defects4J is as follows:

defects4j
|
Expand All @@ -173,20 +205,7 @@ The directory structure is as follows:
|
|--- test: Scripts to test the framework.

Additional resources
--------------------
#### Fault localization (FL)
- [Scripts and annotations for evaluating FL techniques][FL-eval]

#### Automated program repair (APR)
- [Scripts and annotations for evaluating APR techniques][APR-eval]
- [Patches generated with the Nopol, jGenProg, and jKali APR systems][APR-patches-spirals]

[fl-eval]: https://bitbucket.org/rjust/fault-localization-data
[APR-eval]: https://github.com/LASER-UMASS/AutomatedRepairApplicabilityData
[APR-patches-spirals]: https://github.com/Spirals-Team/defects4j-repair

License
---------
MIT License, see `license.txt` for more information.

MIT License, see [`license.txt`](https://github.com/rjust/defects4j/blob/master/license.txt) for more information.
36 changes: 11 additions & 25 deletions framework/test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,41 +32,27 @@ defects (see the top-level
[README](https://github.com/rjust/defects4j/blob/master/README.md) for
more details about the defects and requirements).

### Getting started
1. Follow the instructions for setting up Defects4J in the top-level
[README](https://github.com/rjust/defects4j/blob/master/README.md#setting-up-defects4j)

2. Verify you have the perl package for DBI access to CSV files installed:
- `perldoc DBD/CSV.pm`

If this produces the man page for DBM::CSV you are ok, if not
you must install the package, e.g., using cpan:
- `cpan DBD::CSV`

### Running the coverage analysis
3. Please note you must use Java 7; Java 8 will cause failures.
1. Optionally, indicate where to find the version of Randoop you wish to test.
- `export TESTGEN_LIB_DIR="path2directory-containing-randoop-current.jar"`

4. Tell the tools which version of Randoop you wish to test:
By default, the system runs version 4.0.3 of Randoop.
(Located at "path2defects4j"/framework/lib/test_generation/generation/randooop-current.jar)
The randoop.jar you wish to test must be named randoop-current.jar.
- `export TESTGEN_LIB_DIR="path2directory-containing-randoop-current.jar"`
By default, the system runs version 4.0.3 of Randoop,
located at "path2defects4j"/framework/lib/test_generation/generation/randooop-current.jar.

5. Run the test generation and coverage analysis:
2. Run the test generation and coverage analysis:
- `./randoop_coverage.sh`

Currently, this does not generate tests for all the defects, just five in
each projects for a total of 30 tests. It takes about 90 minutes to run.
each project for a total of 30 tests. It takes about 90 minutes to run.
If you wish to override these defaults for `randoop_coverage.sh` you may
supply an optional project list argument followed by an optional bid list
argument.
supply an optional project list argument followed by an optional bid
(bug id) list argument.
The test scripts set `TMP_DIR` to */tmp/test_d4j*. If you wish to change
this, you will need to modify `./test.include`.

6. Display the coverage data:
The raw coverage data will be found at *$TMP_DIR/test_d4j/coverage*.
You may display the coverage results by running the perl script:
3. Display the coverage data:
- `../util/show_coverage.pl`

The raw coverage data is found at *$TMP_DIR/test_d4j/coverage*.
This script will accept an optional argument of an alternative file location.
Invoke the script with -help for a full list of options.
Invoke the script with `-help` for a full list of options.