Repository for subjects related to the Dependent Tests Impact project

README.md

This file explains how to reproduce figures 5, 10, 11, and 12 in our paper.

You need:

  • Java 7 (the version of Soot that we use does not work with Java 8). We ran our experiments on a machine with Java version "1.7.0_91".
  • Ant
  • Maven

For simplicity, we highly recommend you use the version of our artifact that is pre-installed in this: virtual machine.

The dependent-tests-impact folder is under /home/user/dependent-tests-impact.

Run the following steps to set up the Ambari project. Running the following steps takes about an hour. (If you are on the VM, you can skip these steps, because they have already been performed.)

cd dependent-tests-impact/experiments/ambari
mvn compile
mvn test-compile
mvn install -fn -DskipTests dependency:copy-dependencies
cd ../ambari-new
mvn compile
mvn test-compile
mvn install -fn -DskipTests dependency:copy-dependencies

[[The mvn install -fn -DskipTests dependency:copy-dependencies step failed for me. See email for details.]]

To reproduce figures 5, 10, 11, and 12 perform the following:

cd dependent-tests-impact/experiments
./figureGenerator.sh

The figureGenerator.sh script takes about 4 hours to complete.

The subject program print some error messages to the console. This is expected and normal as some tests do fail and/or produce errors.

Once the script finishes running, the results for prioritization, selection and parallelization can be found in .tex files in directories:

dependent-tests-impact/experiments/
dependent-tests-impact/experiments/prioritization-results/
dependent-tests-impact/experiments/selection-results/
dependent-tests-impact/experiments/parallelization-results/

Since figures 10, 11, and 12 depend on the execution time of the tests, the results may not be identical to those in the submitted paper. While the numerical results differ, they still support the paper's claims. The results used to generate figure 5 in our paper can be found in the dependent-tests-impact/results/ directory. The results used to generate figures 10, 11, and 12 can be found in the dependent-tests-impact/results/issta17/enhanced-figures/results-in-paper/ directory.

The precomputed dependences can be found in directories dependent-tests-impact/experiments/prioritization-dt-list, dependent-tests-impact/experiments/parallelization-dt-list, and dependent-tests-impact/experiments/selection-dt-list, respectively. You can re-generate them if you want, but it may take up to 200 hours.

[[Turn the instructions into a script or Maven target, rather than requiring a reader to follow complex, error-prone instructions.]]

To re-generate them:

  • delete the files in the directories listed above, and
  • uncomment line 32 in prioritization-runner.sh, line 28 in newExperimentsPrioritizationRunner.sh, line 69 in selection-runner.sh, line 50 in newExperimentsSelectionRunner.sh, lines 154 and 156 in config.sh, and lines 46 and 48 in newExp-config.sh (#java ... -> java ...)

However, note that doing so may take 200 hours.

The detailed results for these pre-computed dependences can be found in /home/user/dependent-tests-impact/results/issta17/fixed-dt-results/results-in-paper. The detailed results include the pre-computed dependences that were generated, the time it took to generate such dependency information, etc. These detailed results were also used to generate Figure 9 in our paper.

Figure 4 is generated from running cloc on the subjects' source and test directories and ExamDiff Pro between the two revisions of our subjects.