FaultLocalizationResearch

How-To Set up

For Mac & Linux

Step 1: Set up Defects4J

Refer https://github.com/rjust/defects4j
Clone the repository
Follow documentation steps
  Error:Can't Locate DBI.pm 
  Resolution for Mac: perl -MCPAN -e 'install DBI'
                Install Postgres if required.
  Resolution for Linux: sudo apt install libdbi-perl

Step 2: Clone the repo https://bitbucket.org/rjust/fault-localization-data/overview

Step 3: Download and install JDK 1.6 and JDK 1.8.

Step 4: Set evnironment variables

The path of defects4j installed in step 1
  export D4J_HOME=/Users/{username}/Downloads/defects4j

The path to this root directory of the cloned repo fault-localization-data and append 'gzoltar/gzoltar.jar'
  export GZOLTAR_JAR=/Users/{username}/Downloads/fault-localization-data/gzoltar/gzoltar.jar 

Set JAVA_HOME to point to JDK1.6 Home if you have a different Java default version
  export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
  
Add to PATH variable
  export PATH=$PATH:$D4J_HOME/framework/bin

Step 6: Replace the run_gzoltar.sh provided in this repository in fault-localization-data/gzoltar/gzoltar

Step 5: Test if set up works

Test defects4j

    defects4j info -p Lang
    
Test Gzoltar 

    `bash run_gzoltar.sh Lang 37 . developer`

Step 7: Install sloccount

For Mac: brew install sloccount
For Mac: export SLOC_HOME=/usr/local/bin/sloccount
For Ubuntu: sudo apt-get install sloccount
For Ubuntu: export SLOC_HOME=/usr/bin/sloccount

Step 8: Run the script to get the buggy code

bash get_fixed_lines.sh Lang 37 .

This creates a file `Lang-37.fixed.lines` which contains the line fixed in the human patch for the corresponding bug.
This can be used to evaluate the suspiciousness score generated by the Fault localization technique.

To manually run junit tests in the defects4j projects

Step 1: checkout defects4j project(example: Lang)

defects4j checkout -p Lang -v 37b -w /tmp/Lang37

Step 2: Download JUNIT Jar

Download the junit jar from https://github.com/downloads/junit-team/junit/junit-4.10.jar

Step 3: Downgrade Maven to mvn 3.2 so as to run with Java 1.6

Check maven version by running.
    mvn --version
If this is 3.2 or below then skip to Step 4.
If not follow these steps to downgrade maven
    brew install maven@3.2
    brew unline maven
    brew link --force --overwrite maven@3.2

Step 4: Compile Lang Project

Go to the checked out Lang project folder in two session on Terminal
Session 1:
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
    mvn compile (This will fail)
Session 2:
    mvn compile
    rm -rf target/*
Session 1:
    mvn compile (This should succeed)

Step 5: Run all of the Junit tests in Lang Project using maven

Session 1:
    mvn -Dmaven.test.failure.ignore=true install (This will fail)
Session 2:
    mvn -Dmaven.test.failure.ignore=true install
    rm -rf target/commons-lang-3.0-SNAPSHOT.jar
Session 1:
    mvn -Dmaven.test.failure.ignore=true install (This should succeed)
    mvn test

Step 6: Run one of the Junit tests in Lang Project from command line without maven

cd target
cp {Path to junit jar}/junit-4.10.jar .
java -cp .:/tmp/Lang37/target/test-classes/:junit-4.10.jar:commons-lang-3.0-SNAPSHOT.jar org.junit.runner.JUnitCore org.apache.commons.lang3.ArrayUtilsAddTest

Step 7a: Setting up javaslicer Clone the repository for javaslicer (https://github.com/hammacher/javaslicer) cd javaslicer ./assemble.sh (Note: There might be failures when you run this the first time. Run ./assemble.sh to fix it.)

Step 7b: Run one of the junit test classes with tracer as a javaagent attached.

Session 2:
    java -cp .:/tmp/Lang37/target/test-classes/:junit-4.10.jar:commons-lang-3.0-SNAPSHOT.jar -javaagent:/Users/jithinjohn/Downloads/CSC591/javaslicer/assembly/tracer.jar=tracefile:test.trace org.junit.runner.JUnitCore org.apache.commons.lang3.ArrayUtilsAddTest

Step 8: Run slicer to produce slicing results

Session 2:
    java -Xmx2g -jar /Users/jithinjohn/Downloads/CSC591/javaslicer/assembly/slicer.jar -p test.trace org.apache.commons.lang3.ArrayUtils.addAll:2962:* > output.txt

Automation

Here are sample commands on how to run the slicer, tracer along with some utilities developed for the automation manually. These are performed by the automation script as specified below.

Change directory to automation/src/

COMPILE:

javac -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests.java

RUN:

1. Running a test file

java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 runTestFile org.apache.commons.lang3.ArrayUtilsAddTest

2. Running a test specific test case

java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 runTestCase org.apache.commons.lang3.ArrayUtilsAddTest testJira567

3. Getting test cases of a test file

java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 getTestCases org.apache.commons.lang3.ArrayUtilsAddTest

4. Getting line numbers of assert statements in a test case

java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 getAssertLines org.apache.commons.lang3.ArrayUtilsAddTest testJira567

Overview

The experiments evaluate various fault localization techniques on artificial faults and on real faults.

At a high level, here's how it all works:

• The real and artificial faults come from the Defects4J Project.
• For each D4J fault, the scripts in d4j_integration/ determine which lines are faulty. The resultant files are "buggy-lines" files, and live in analysis/pipeline-scripts/buggy-lines/.
• Many fault localization techniques require coverage information. We use GZoltar and Tacoco to gather coverage information. The resultant files are called "matrix", "spectra" and {defectId}-cov-matrix.json.
• Our scripts enable you to compute all the suspiciousness score and coverage information, but doing so takes a great deal of computation.

How to generate scores

• To run Test Case Purification and Rank refinement, use automation/automate.py.

  Example invocation:

  python3 automate.py \
    Lang 37 \
    /tmp/Lang-37 
  

  Creates the files Lang-37.killmap.csv.gz and Lang-37.mutants.log.

• To run GZoltar seperately, use gzoltar/run_gzoltar.sh.

  Example invocation: bash run_gzoltar.sh Lang 37 . developer

  Creates the files ./matrix and ./spectra.

Contents

• automation/: Scripts and tools required to run analysis on the bugs in defects4j.

• d4j_integration/: Scripts that build upon or extend Defects4J to populate or query its database.

• gzoltar/: Scripts for running the GZoltar tool to collect line coverage information.

=======

Results in Work Directory

If the automate.py script was invoked with /tmp/LangTest as work directory for example say with Project as Lang and Bug id 37 then the results in the word directory would look like this.

`tmp/LangTest`: Root Working directory all results are under this main directory.
│
├──`Lang/`: Any bugs from Lang Project would checked out using defects4j under this folder.
│   │
│   ├──`37/`: Bug ID - 37 is checked out here, then compiled and packaged to the Jar.
│
├──`gzoltars/`: All results generated by gzoltar is this directory.
│   │
│   ├──`Lang/`: Results of gzoltar for any bugs from Lang Project would created here.
│   │   │
│   │   ├──`37/`: Gzoltar results for Bug Id 37 would be present here including the coverage matrix, spectra.
│
├──`tacocos/`: Results of the tacoco tool and the final results of the tool would be here.
│   │
│   ├──`Lang/`: Results for all bugs in Lang Project
│
├──`traces/`: Any bugs from Lang Project would checked out using defects4j under this folder.
│   │
│   ├──`Lang/`: Results tracer and slicer for all bugs in Lang Project
│   │   │
│   │   ├──`37/`: Tracer and slicer output files for bug id 37 in Lang Project .