Permalink
Browse files

Improve documentation (#156)

* Improve documentation

* Reorganization of documentation
  • Loading branch information...
mernst authored and rjust committed May 20, 2018
1 parent 486e2b4 commit 500066036432ed48d1ffc82255d2cd9fa195e84e
Showing with 96 additions and 91 deletions.
  1. +85 −66 README.md
  2. +11 −25 framework/test/README.md
View
151 README.md
@@ -1,8 +1,11 @@
Defects4J -- version 1.2.0 [![Build Status](https://travis-ci.org/rjust/defects4j.svg?branch=master)](https://travis-ci.org/rjust/defects4j)
----------------
================
Defects4J is a collection of reproducible bugs and a supporting infrastructure
with the goal of advancing software engineering research.
Contents of Defects4J
================
The projects
---------------
Defects4J contains 395 bugs from the following open-source projects:
@@ -22,8 +25,9 @@ Each bug has the following properties:
- Issue filed in the corresponding issue tracker, and issue tracker identifier
mentioned in the fixing commit message.
- Fixed in a single commit -- the Defects4J maintainers manually pruned out
irrelevant changes (e.g., refactorings or feature additions).
- Fixed in a single commit
- Minimized: the Defects4J maintainers manually pruned out
irrelevant changes in the commit (e.g., refactorings or feature additions).
- Fixed by modifying the source code (as opposed to configuration files,
documentation, or test files).
- A triggering test exists that failed before the fix and passes after the fix
@@ -32,11 +36,14 @@ Each bug has the following properties:
The (b)uggy and (f)ixed program revisions are labelled with `<id>b` and
`<id>f`, respectively (`<id>` is an integer).
Setting up Defects4J
================
Requirements
----------------
- Java 1.7
- Perl >= 5.0.10
- Perl modules: run `cpan Bundle::CSV DBI` or `sudo cpan Bundle::CSV DBI`
- Perl modules: run `cpan Bundle::CSV DBD::CSV DBI` or `sudo cpan Bundle::CSV DBD::CSV DBI`
- Git >= 1.9
- SVN >= 1.8
@@ -50,9 +57,9 @@ Defects4J generates and executes tests in the timezone `America/Los_Angeles`.
If you are using the bugs outside of the Defects4J framework, export the `TZ`
environment variable accordingly.
Getting started
Steps to set up Defects4J
----------------
#### Setting up Defects4J
1. Clone Defects4J:
- `git clone https://github.com/rjust/defects4j`
@@ -63,57 +70,45 @@ Getting started
3. Add Defects4J's executables to your PATH:
- `export PATH=$PATH:"path2defects4j"/framework/bin`
#### Using Defects4J
4. Check installation and get information for a specific project (commons lang):
4. Check installation:
- `defects4j info -p Lang`
Using Defects4J
================
#### Example commands
1. Get information for a specific project (commons lang):
- `defects4j info -p Lang`
5. Get information for a specific bug (commons lang, bug 1):
2. Get information for a specific bug (commons lang, bug 1):
- `defects4j info -p Lang -b 1`
6. Checkout a buggy source code version (commons lang, bug 1, buggy version):
3. Checkout a buggy source code version (commons lang, bug 1, buggy version):
- `defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy`
7. Change to the working directory, compile sources and tests, and run tests:
4. Change to the working directory, compile sources and tests, and run tests:
- `cd /tmp/lang_1_buggy`
- `defects4j compile`
- `defects4j test`
8. More examples of how to use the framework are available in `framework/test`
5. The scripts in [`framework/test/`](tree/master/framework/test/)
are examples of how to use Defects4J, which you might find useful
as inspiration when you are writing your own scripts that use Defects4J.
Publications
------------------
* "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs"
René Just, Darioush Jalali, and Michael D. Ernst,
ISSTA 2014 [[download]][issta14].
* "Are Mutants a Valid Substitute for Real Faults in Software Testing?"
René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser,
FSE 2014 [[download]][fse14].
[issta14]: https://people.cs.umass.edu/~rjust/publ/defects4j_issta_2014.pdf
[fse14]: https://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf
Documentation
--------------------
Detailed documentation for any script or module is available as
[html documentation][htmldocs].
[htmldocs]: http://people.cs.umass.edu/~rjust/defects4j/html_doc/index.html
Command-line interface
Command-line interface: defects4j command
-----------------------
Use `framework/bin/defects4j` to execute any of the following commands:
Use [`framework/bin/defects4j`](http://people.cs.umass.edu/~rjust/defects4j/html_doc/defects4j.html) to execute any of the following commands:
| Command | Description |
|----------------|---------------------------------------------------------------------------------------------------|
| info | View configuration of a specific project or summary of a specific bug |
| checkout | Checkout a buggy or a fixed project version |
| compile | Compile sources and developer-written tests of a buggy or a fixed project version |
| test | Run a single test method or a test suite on a buggy or a fixed project version |
| mutation | Run mutation analysis on a buggy or a fixed project version |
| coverage | Run code coverage analysis on a buggy or a fixed project version |
| monitor.test | Monitor the class loader during the execution of a single test or a test suite |
| export | Export version-specific properties such as classpaths, directories, or lists of tests |
| [info](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-info.html) | View configuration of a specific project or summary of a specific bug |
| [checkout](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-checkout.html) | Checkout a buggy or a fixed project version |
| [compile](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-compile.html) | Compile sources and developer-written tests of a buggy or a fixed project version |
| [test](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-test.html) | Run a single test method or a test suite on a buggy or a fixed project version |
| [mutation](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-mutation.html) | Run mutation analysis on a buggy or a fixed project version |
| [coverage](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-coverage.html) | Run code coverage analysis on a buggy or a fixed project version |
| [monitor.test](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-monitor.test.html) | Monitor the class loader during the execution of a single test or a test suite |
| [export](http://people.cs.umass.edu/~rjust/defects4j/html_doc/d4j/d4j-export.html) | Export version-specific properties such as classpaths, directories, or lists of tests |
Export version-specific properties
@@ -140,17 +135,54 @@ provides the following scripts:
| Script | Description |
|-------------------|-----------------------------------------------------------------|
| run_bug_detection | ^Determine the real fault detection rate |
| run_mutation | ^Determine the mutation score |
| run_coverage | ^Determine code coverage ratios (statement and branch coverage) |
| run_evosuite | Generate test suites using EvoSuite |
| run_randoop | Generate test suites using Randoop |
^Note that this script requires Perl DBI.
Directory structure
| [defects4j](http://people.cs.umass.edu/~rjust/defects4j/html_doc/defects4j.html) | Main script, described above |
| [run_bug_detection](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_bug_detection.html) | Determine the real fault detection rate |
| [run_mutation](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_mutation.html) | Determine the mutation score |
| [run_coverage](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_coverage.html) | Determine code coverage ratios (statement and branch coverage) |
| [run_evosuite](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_evosuite.html) | Generate test suites using EvoSuite |
| [run_randoop](http://people.cs.umass.edu/~rjust/defects4j/html_doc/run_randoop.html) | Generate test suites using Randoop |
Additional resources
================
Scripts built on Defects4J
--------------------
#### Fault localization (FL)
- [Scripts and annotations for evaluating FL techniques][FL-eval]
#### Automated program repair (APR)
- [Scripts and annotations for evaluating APR techniques][APR-eval]
- [Patches generated with the Nopol, jGenProg, and jKali APR systems][APR-patches-spirals]
[fl-eval]: https://bitbucket.org/rjust/fault-localization-data
[APR-eval]: https://github.com/LASER-UMASS/AutomatedRepairApplicabilityData
[APR-patches-spirals]: https://github.com/Spirals-Team/defects4j-repair
Publications
------------------
* "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs"
René Just, Darioush Jalali, and Michael D. Ernst,
ISSTA 2014 [[download]][issta14].
* "Are Mutants a Valid Substitute for Real Faults in Software Testing?"
René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser,
FSE 2014 [[download]][fse14].
[issta14]: https://people.cs.umass.edu/~rjust/publ/defects4j_issta_2014.pdf
[fse14]: https://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf
[More publications](https://scholar.google.com/scholar?q=defects4j)
Implementation details
----------------------
This is the top-level directory of Defects4J.
The directory structure is as follows:
Documentation for any script or module is available as
[html documentation][htmldocs].
[htmldocs]: http://people.cs.umass.edu/~rjust/defects4j/html_doc/index.html
The directory structure of Defects4J is as follows:
defects4j
|
@@ -173,20 +205,7 @@ The directory structure is as follows:
|
|--- test: Scripts to test the framework.
Additional resources
--------------------
#### Fault localization (FL)
- [Scripts and annotations for evaluating FL techniques][FL-eval]
#### Automated program repair (APR)
- [Scripts and annotations for evaluating APR techniques][APR-eval]
- [Patches generated with the Nopol, jGenProg, and jKali APR systems][APR-patches-spirals]
[fl-eval]: https://bitbucket.org/rjust/fault-localization-data
[APR-eval]: https://github.com/LASER-UMASS/AutomatedRepairApplicabilityData
[APR-patches-spirals]: https://github.com/Spirals-Team/defects4j-repair
License
---------
MIT License, see `license.txt` for more information.
MIT License, see [`license.txt`](https://github.com/rjust/defects4j/blob/master/license.txt) for more information.
View
@@ -32,41 +32,27 @@ defects (see the top-level
[README](https://github.com/rjust/defects4j/blob/master/README.md) for
more details about the defects and requirements).
### Getting started
1. Follow the instructions for setting up Defects4J in the top-level
[README](https://github.com/rjust/defects4j/blob/master/README.md#setting-up-defects4j)
2. Verify you have the perl package for DBI access to CSV files installed:
- `perldoc DBD/CSV.pm`
If this produces the man page for DBM::CSV you are ok, if not
you must install the package, e.g., using cpan:
- `cpan DBD::CSV`
### Running the coverage analysis
3. Please note you must use Java 7; Java 8 will cause failures.
1. Optionally, indicate where to find the version of Randoop you wish to test.
- `export TESTGEN_LIB_DIR="path2directory-containing-randoop-current.jar"`
4. Tell the tools which version of Randoop you wish to test:
By default, the system runs version 4.0.3 of Randoop.
(Located at "path2defects4j"/framework/lib/test_generation/generation/randooop-current.jar)
The randoop.jar you wish to test must be named randoop-current.jar.
- `export TESTGEN_LIB_DIR="path2directory-containing-randoop-current.jar"`
By default, the system runs version 4.0.3 of Randoop,
located at "path2defects4j"/framework/lib/test_generation/generation/randooop-current.jar.
5. Run the test generation and coverage analysis:
2. Run the test generation and coverage analysis:
- `./randoop_coverage.sh`
Currently, this does not generate tests for all the defects, just five in
each projects for a total of 30 tests. It takes about 90 minutes to run.
each project for a total of 30 tests. It takes about 90 minutes to run.
If you wish to override these defaults for `randoop_coverage.sh` you may
supply an optional project list argument followed by an optional bid list
argument.
supply an optional project list argument followed by an optional bid
(bug id) list argument.
The test scripts set `TMP_DIR` to */tmp/test_d4j*. If you wish to change
this, you will need to modify `./test.include`.
6. Display the coverage data:
The raw coverage data will be found at *$TMP_DIR/test_d4j/coverage*.
You may display the coverage results by running the perl script:
3. Display the coverage data:
- `../util/show_coverage.pl`
The raw coverage data is found at *$TMP_DIR/test_d4j/coverage*.
This script will accept an optional argument of an alternative file location.
Invoke the script with -help for a full list of options.
Invoke the script with `-help` for a full list of options.

0 comments on commit 5000660

Please sign in to comment.