Skip to content

Commit

Permalink
Test gen (#186)
Browse files Browse the repository at this point in the history 
* Added a generic test-generation script.
* Added wrapper scripts for Randoop and EvoSuite.
* Made test mode (regression vs. error-revealing) a parameter.
* Export list of relevant (loaded) classes to d4j properties file.
* Added script to test test generation for arbitrary bugs
* Removed test-gen functionality from Project module.

Co-authored-by: Michael Ernst <mernst@alum.mit.edu>
Co-authored-by: Gregory Gay <greg@greggay.com>
  • Loading branch information
@rjust @mernst @Greg4cr
3 people committed Feb 15, 2020
1 parent cdf5fe6 commit 53bea06
Show file tree
Hide file tree
Showing 23 changed files with 848 additions and 749 deletions.
3 changes: 2 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
framework/lib/analyzer.jar
framework/lib/build-analyzer-*.jar
framework/lib/build_systems
framework/lib/test_generation
framework/lib/test_generation/*
!framework/lib/test_generation/bin
framework/test/*.log
project_repos/*
!project_repos/get_repos.sh
Expand Down
6 changes: 2 additions & 4 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,8 @@ jobs:
name: test_tutorial.sh
- script: carton exec ./test_mutation_analysis.sh
name: test_mutation_analysis.sh
- script: carton exec ./test_randoop.sh
name: test_randoop.sh
- script: carton exec ./test_evosuite.sh
name: test_evosuite.sh
- script: carton exec ./test_gen_tests.sh -p Lang -b 6
name: test_gen_tests.sh
- script: carton exec ./test_fix_test_suite.sh
name: test_fix_test_suite.sh
- script: carton exec ./test_export_command.sh
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,8 @@ directory to export a version-specific property:

| Property | Description |
|------------------|-------------------------------------------------------------------------------------|
| classes.modified | Classes (source files) modified by the bug fix |
| classes.modified | Classes modified by the bug fix |
| classes.relevant | Classes loaded by the JVM when executing all triggering tests |
| cp.compile | Classpath to compile and run the project |
| cp.test | Classpath to compile and run the developer-written tests |
| dir.src.classes | Source directory of classes (relative to working directory) |
Expand All @@ -176,11 +177,10 @@ provides the following scripts:
| Script | Description |
|-------------------|-----------------------------------------------------------------|
| [defects4j](http://defects4j.org/html_doc/defects4j.html) | Main script, described above |
| [gen_tests](http://defects4j.org/html_doc/gen_tests.html) | Generate test suites using EvoSuite or Randoop |
| [run_bug_detection](http://defects4j.org/html_doc/run_bug_detection.html) | Determine the real fault detection rate |
| [run_mutation](http://defects4j.org/html_doc/run_mutation.html) | Determine the mutation score |
| [run_coverage](http://defects4j.org/html_doc/run_coverage.html) | Determine code coverage ratios (statement and branch coverage) |
| [run_evosuite](http://defects4j.org/html_doc/run_evosuite.html) | Generate test suites using EvoSuite |
| [run_randoop](http://defects4j.org/html_doc/run_randoop.html) | Generate test suites using Randoop |

Mining and contributing additional bugs to Defects4J
================
Expand Down
307 changes: 307 additions & 0 deletions framework/bin/gen_tests.pl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,307 @@
#!/usr/bin/env perl
#
#-------------------------------------------------------------------------------
# Copyright (c) 2014-2020 René Just, Darioush Jalali, and Defects4J contributors.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#-------------------------------------------------------------------------------

=pod
=head1 NAME
gen_tests.pl -- generate a test suite using one of the supported test generators.
=head1 SYNOPSIS
gen_tests.pl -g generator -p project_id -v version_id -n test_id -o out_dir -b total_budget [-c target_classes] [-s random_seed] [-t tmp_dir] [-E] [-D]
=head1 OPTIONS
=over 4
=item -g C<generator>
The test generator to use. Run the following command to see a list of supported
test generators: C<run_test_gen.pl -g help>
=item -p C<project_id>
Generate tests for this project id.
See L<Project|Project/"Available Project IDs"> module for available project IDs.
=item -v C<version_id>
Generate tests for this version id.
Format: C<\d+[bf]>.
=item -n C<test_id>
The id of the generated test suite (i.e., which run of the same configuration).
=item -o F<out_dir>
The root output directory for the generated test suite. The test suite and logs are
written to:
F<out_dir/project_id/version_id>.
=item -b C<total_budget>
The total time in seconds allowed for test generation.
=item -c F<classes_file>
The file that lists all classes the test generator should target, one class per
line (optional). By default, tests are generated only for classes modified by
the bug fix.
=item -s C<random_seed>
The random seed used for test generation (optional). By default, the random seed
is computed as: <test_id> * 1000 + <bug_id>.
=item -t F<tmp_dir>
The temporary root directory to be used to check out the program version (optional).
The default is F</tmp>.
=item -E
Generate error-revealing (as opposed to regression) tests (optional).
By default this script generates regression tests, regardless of whether the
project version is a buggy or a fixed project version.
Note that not all test generators support both modes (i.e., generating
regression tests and generating error-revealing tests).
=item -D
Debug: Enable verbose logging and do not delete the temporary check-out directory
(optional).
=back
=head1 DESCRIPTION
This script runs the specified test generator on a particular program version.
Tests are, by default, generated for all classes modified by the bug fix; a set
of target classes can be specified using the C<-c> flag.
=head2 Tool configuration
The following wrapper script invokes the specified test generator and provides
the generator-specific configuration:
F<C<TESTGEN_BIN_DIR>/C<GENERATOR>.sh>.
=cut
use strict;
use warnings;

use FindBin;
use File::Basename;
use Cwd qw(abs_path);
use Getopt::Std;
use Pod::Usage;

use lib abs_path("$FindBin::Bin/../core");
use Constants;
use Utils;
use Project;
use Log;

#
# Process arguments and issue usage message if necessary.
#
my %cmd_opts;
getopts('g:p:v:o:n:b:c:s:t:ED', \%cmd_opts) or pod2usage(1);
my $TOOL = $cmd_opts{g};
# Print all supported generators, regardless of the other arguments, if -g help
# is set
defined $TOOL and $TOOL =~ /^help$/ and is_generator_valid($TOOL) || exit 1;

pod2usage(1) unless defined $cmd_opts{g} and
defined $cmd_opts{p} and
defined $cmd_opts{v} and
defined $cmd_opts{n} and
defined $cmd_opts{b} and
defined $cmd_opts{o};

# Check whether the requested test generator is valid
is_generator_valid($TOOL) || exit 1;

my $PID = $cmd_opts{p};
# Instantiate project
my $project = Project::create_project($PID);

my $VID = $cmd_opts{v};
# Verify that the provided version id is valid
my $BID = Utils::check_vid($VID)->{bid};
$project->contains_version_id($VID) or die "Version id ($VID) does not exist in project: $PID";

# Verify that the provided test id is valid
my $TID = $cmd_opts{n};
$TID =~ /^\d+$/ or die "Wrong test_id format (\\d+): $TID!";

# Verify that the provided time budget is valid
my $TIME = $cmd_opts{b};
$TIME =~ /^\d+$/ or die "Wrong budget format (\\d+): $TIME!";

my $OUT_DIR = $cmd_opts{o};

# Set or compute the random seed
my $SEED = $cmd_opts{s} // $TID*1000 + $BID;

# List of target classes (list of modified classes is the default)
my $TARGET_CLASSES = $cmd_opts{c} // "$SCRIPT_DIR/projects/$PID/modified_classes/$BID.src";

# Generate regression tests by default
my $MODE = (defined $cmd_opts{E}) ? "error-revealing" : "regression";

# Enable debugging if flag is set
$DEBUG = 1 if defined $cmd_opts{D};

# Temporary directory for project checkout
my $TMP_DIR = Utils::get_tmp_dir($cmd_opts{t});
system("mkdir -p $TMP_DIR");

# Set working directory
$project->{prog_root} = $TMP_DIR;

=pod
=head2 Logging
By default, the script logs all errors and warnings to F<gen_tests.log> in
the temporary project root.
Upon success, the log of this script is appended to:
F<out_dir/logs/C<project_id>.C<version_id>.log>.
=cut
# Log file in output directory
my $LOG_DIR = "$OUT_DIR/logs";
my $LOG_FILE = "$LOG_DIR/$PID.$VID.log";
system("mkdir -p $LOG_DIR");

# Checkout and compile project
$project->checkout_vid($VID) or die "Cannot checkout!";
$project->compile() or die "Cannot compile!";

# Open temporary log file
my $LOG = Log::create_log("$TMP_DIR/$PID.$VID.$TID.log");
$LOG->log_time("Start test generation");
$LOG->log_msg("Mode: $MODE");
$LOG->log_msg("Parameters:");
$LOG->log_msg(" -g $TOOL");
$LOG->log_msg(" -p $PID");
$LOG->log_msg(" -v $VID");
$LOG->log_msg(" -n $TID");
$LOG->log_msg(" -b $TIME");
$LOG->log_msg(" -c $TARGET_CLASSES");
$LOG->log_msg(" -s $SEED");

# Export all environment variables that are expected by the wrapper script of
# the test generator.
$ENV{D4J_HOME} = "$BASE_DIR";
$ENV{D4J_FILE_TARGET_CLASSES} = "$TARGET_CLASSES";
$ENV{D4J_DIR_WORKDIR} = "$TMP_DIR";
$ENV{D4J_DIR_OUTPUT} = "$TMP_DIR/$TOOL";
$ENV{D4J_DIR_LOG} = "$LOG_DIR";
$ENV{D4J_DIR_TESTGEN_BIN} = "$TESTGEN_BIN_DIR";
$ENV{D4J_DIR_TESTGEN_LIB} = "$TESTGEN_LIB_DIR";
$ENV{D4J_TOTAL_BUDGET} = "$TIME";
$ENV{D4J_SEED} = "$SEED";
$ENV{D4J_TEST_MODE} = "$MODE";

# Invoke the test generator
Utils::exec_cmd("$TESTGEN_BIN_DIR/$TOOL.sh", "Generating tests ($TOOL)")
or die "Failed to generate tests!";

# Compress generated tests and copy archive to output directory
my $archive = "$PID-$VID-$TOOL.$TID.tar.bz2";
Utils::exec_cmd("tar -cjf $TMP_DIR/$archive -C $TMP_DIR/$TOOL/ .", "Creating test suite archive")
or die("Cannot archive and compress test suite!");

$LOG->log_msg("Created test suite archive: $archive");

# Acknowledge the tool author(s)
system("cat $TESTGEN_BIN_DIR/$TOOL.credit");

=pod
=head2 Test suites
The source files of the generated test suite are compressed into an archive with
the following name:
F<C<project_id>-C<version_id>-C<tool>.C<test_id>.tar.bz2>
Examples:
=over 4
=item * F<Lang-12b-randoop.1.tar.bz2>
=item * F<Lang-12f-randoop.2.tar.bz2>
=item * F<Lang-12f-evosuite.1.tar.bz2>
=back
The test suite archive is written to:
F<out_dir/C<project_id>/C<TOOL>/C<test_id>>
=cut

# Move test suite to OUT_DIR/pid/suite_src/test_id
#
# e.g., .../Lang/randoop/1
# .../Lang/evosuite/1
#
my $dir = "$OUT_DIR/$PID/$TOOL/$TID";
system("mkdir -p $dir && mv $TMP_DIR/$archive $dir") == 0
or die "Cannot move test suite archive to output directory!";

$LOG->log_msg("Moved test suite archive to output directory: $dir");

$LOG->log_time("End test generation");

# Close temporary log and append content to log file in output directory
$LOG->close();
system("cat $LOG->{file_name} >> $LOG_FILE");

# Remove temporary directory
system("rm -rf $TMP_DIR") unless $DEBUG;

################################################################################
# Check whether the requested generator is valid
sub is_generator_valid {
my $tool = shift;
my %all_tools;
opendir(my $dir, "$TESTGEN_BIN_DIR") || die("Cannot read test generators: $!");
while (readdir $dir) {
/^([^_].+)\.sh/ and $all_tools{$1} = 1;
}
closedir $dir;
unless(defined $all_tools{$tool}) {
print("Supported test generators:\n");
foreach (sort keys %all_tools) {
print("- $_\n");
}
return 0;
}
return 1;
}
Loading

0 comments on commit 53bea06

Please sign in to comment.