flacocobot support for fault-localization in Repairnator (eclipse#1243

)

Co-authored-by: Davide Ginelli <ginellidavide@gmail.com>

.gitignore

@@ -36,3 +36,5 @@ repairnator/repairnator-pipeline/spooned*
 repairnator/repairnator-pipeline/output_astor
 /bin/
 *.java-version
+
+.DS_Store

doc/README.md

@@ -13,6 +13,7 @@ Repairnator can be used:
 * [as a Github app](https://github.com/eclipse/repairnator/blob/master/doc/repairnator-github-app.md)
 * as a command line tool, see "Command line" below
 * as a Travis CI scanner, see "Travis Scanner" below
+* as Pull Requests scanner with Flacocobot, see "Flacoco Scanner" below
 
 ## Contributing
 
@@ -46,6 +47,19 @@ cd repairnator/src/scripts/
 bash launch_rtscanner.sh
 ```
 
+### Flacoco Scanner
+
+Prerequisites: Java, Docker
+
+```
+git clone https://github.com/eclipse/repairnator/
+cd repairnator/src/repairnator-realtime
+mvn clean package -DskipTests
+java -cp target/repairnator-realtime-<version>-jar-with-dependencies.jar fr.inria.spirals.repairnator.realtime.FlacocoScanner
+```
+
+The documentation is available [here](main-classes.md#flacoco-scanner).
+
 ## Program repair tools used in Repairnator
 
 For more information about the program repair tools and their strategies implemented in Repairnator, [take a look at this page](repair-tools.md).

doc/main-classes.md

@@ -193,15 +193,17 @@ java -cp $TOOLS_JAR:target/repairnator-pipeline-3.3-SNAPSHOT-jar-with-dependenci
 ### Fault localization mode
 
 Repairnator also features a fault localization mode. 
-In this mode, Travis CI builds are analyzed, and the identified suspicious lines are pushed as a PR review/suggestions to the original PR.
+In this mode, a git repository is analyzed, and the identified suspicious lines are pushed as a PR review/suggestions to the original PR.
+
+Note: the personal access token **MUST** have the scope **public_repo** to be able to create pull request reviews.
 
 The example below shows the usage of this mode:
 
 ```bash
 export M2_HOME=/usr/share/maven
 export GITHUB_TOKEN=foobar # your token
 
-java -cp target/repairnator-pipeline-3.3-SNAPSHOT-jar-with-dependencies.jar fr.inria.spirals.repairnator.pipeline.Launcher --launcherMode FAULT_LOCALIZATION --faultLocalization --build 236072272 --flacocoThreshold 0.12 --ghOauth $GITHUB_TOKEN
+java -cp target/repairnator-pipeline-3.3-SNAPSHOT-jar-with-dependencies.jar fr.inria.spirals.repairnator.pipeline.Launcher --launcherMode FAULT_LOCALIZATION --faultLocalization --gitrepourl "https://github.com/andre15silva/failingProject" --gitrepoidcommit 54f241129e09d71955b7d2f4fc7f496118b3e1c6 --flacocoThreshold 0.12 --ghOauth $GITHUB_TOKEN
 ```
 
 The result of running this example can be found [here](https://github.com/repairnator/failingProject/pull/7).
@@ -352,6 +354,63 @@ Options:
 
 ```
 
+## Flacoco Scanner
+
+Flacoco Scanner is the scanner used by Flacocobot (a bot for fault localization that uses [Flacoco](https://github.com/SpoonLabs/flacoco)) to analyze the pull requests associated with the projects under analysis by the bot. This scanner is designed to work in two ways:
+
+1. Add a review comment to the failing pull requests if Flacoco finds suspicious lines contained in the diff introduced by the pull request compared to the main code base (it suggests the top 5 most suspicious lines by default);
+2. Save the fault localization result (the top 5 most suspicious lines in the diff or the top 5 most suspicious lines in general) in a GitHub repository.
+
+To specify the list of projects to be scanned, it is necessary to create a file, add the slug associated with the projects (e.g., eclipse/repairnator, 
+and not https://github.com/eclipse/repairnator), and associate the file path to a system variable called `PROJECTS_TO_SCAN_FILE`. You have to put one project per line. This means that the content of the file with the list of projects to be scanned must have this structure:
+
+```
+<user>/<repo-name>
+<user>/<repo-name>
+<user>/<repo-name>
+...
+```
+
+Flacoco Scanner scans each project every 15 minutes for a total of 14 days by default. At the first iteration, the scanner searches for the failing open pull requests that have been opened in the last 7 days by default. In the following iterations, the scanner listens for every new failing pull request and updates (only in relation with the source code) related to the pull requests previously analyzed.
+
+To customize the scan period, the total time of execution and the number of days before the current date to select only the pull requests opened in that period, you can set these system variables:
+
+```
+FLACOCOBOT_SCAN_INTERVAL // Minutes
+FLACOCOBOT_EXECUTION_TIME // Days
+FLACOCOBOT_CHECK_PR_DAYS_BEFORE_CURRENT_DATE // Days
+```
+
+To set the threshold used by Flacoco, it is necessary to set its value in a system variable called `FLACOCO_THRESHOLD`.
+
+To save the results in a repository instead of adding a review comment to a pull request, it is necessary to create a system variable called `FLACOCO_RESULTS_REPOSITORY` and associate the slug of the repository where results will be saved.
+
+### Example of use
+
+```
+export M2_HOME=/usr/share/maven
+export GITHUB_TOKEN=foobar # Your Token
+export PROJECTS_TO_SCAN_FILE=/path/to/file.txt
+export FLACOCO_RESULTS_REPOSITORY=<user>/<repo-name> # Only to save the results in a repository
+export FLACOCOBOT_SCAN_INTERVAL=60 # Minutes
+export FLACOCOBOT_EXECUTION_TIME=10 # Days
+export FLACOCOBOT_CHECK_PR_DAYS_BEFORE_CURRENT_DATE=30 # Days
+
+git clone https://github.com/eclipse/repairnator/
+cd repairnator
+mvn clean install -DskipTests -f src/repairnator-core/ && mvn clean install -DskipTests -f src/repairnator-pipeline/
+cd src/repairnator-realtime
+mvn clean package -DskipTests
+java -cp target/repairnator-realtime-<version>-jar-with-dependencies.jar fr.inria.spirals.repairnator.realtime.FlacocoScanner --ghOauth $GITHUB_TOKEN
+```
+
+If the variable `FLACOCO_RESULTS_REPOSITORY` has been set with a GitHub repository slug and Flacoco finds suspicious lines, Flacocobot will add the results to that repository.
+If the variable `FLACOCO_RESULTS_REPOSITORY` has not been set with a GitHub repository slug and Flacoco finds suspicious lines, it will add a review comment to the failing pull request.
+
+In the first execution mode, the review comment will be created only if Flacoco finds suspicious lines that are contained in the diff introduced by the failing pull request compared to the main code base.
+
+In the second execution mode, the fault localization results will be saved in specific .MD files in the repository indicated by the variable `LACOCO_RESULTS_REPOSITORY`. If the file name starts with diff_, it means that the file contains the suspicious lines found by Flacocobot that are contained in the diff introduced by the pull request compared to the main code base. Otherwise, the file contains the top 5 most suspicious lines found by Flacoco.
+
 ## fr.inria.spirals.repairnator.dockerpool.BuildAnalyzerLauncher
 
 (internal usage only)

src/docker-images/pipeline-dockerimage/Dockerfile

@@ -1,4 +1,4 @@
-FROM maven:3.6.1-jdk-8
+FROM maven:3.6.1-jdk-11
 
 LABEL Description="Repairnator Pipeline docker image" Vendor="Spirals" Version="0.0.0"
 
@@ -17,7 +17,8 @@ RUN echo `xmlstarlet sel -t -v '//latest' maven-metadata.xml` > /root/version.in
 RUN chmod +x /root/*.sh
 RUN echo "Europe/Paris" > /etc/timezone 
 RUN /root/configure_git.sh 
-RUN /root/build_repairnator.sh
+RUN /root/build_repairnator.sh # This script builds the latest version available on central.
+#COPY repairnator-pipeline.jar /root/repairnator-pipeline.jar # If you want to bypass it, and use a built jar, uncomment this line and comment the one above.
 
 ENV M2_HOME=$MAVEN_HOME
 ENV GITHUB_OAUTH=

src/docker-images/pipeline-dockerimage/pipeline_launcher.sh

@@ -56,7 +56,10 @@ LOCAL_REPAIR_MODE=repair
 
 
 # Github XOR Travis
-if [[ -n "$GITHUB_URL" ]] && [[ -n "$GITHUB_SHA" ]]; then
+if [[ -n "$GITHUB_URL" ]] && [[ -n "$GITHUB_PR" ]]; then
+    echo "adding GitHub PR mode variables"
+    args="$args --gitrepourl $GITHUB_URL --gitrepopullrequest $GITHUB_PR"
+elif [[ -n "$GITHUB_URL" ]] && [[ -n "$GITHUB_SHA" ]]; then
   echo "adding GitHub mode variables"
   args="$args --gitrepourl $GITHUB_URL --gitrepoidcommit $GITHUB_SHA"
 elif [[ -n "$BUILD_ID" ]]; then
@@ -111,4 +114,10 @@ export GITHUB_USEREMAIL=
 
 echo "Execute pipeline with following supplementary args: $args"
 
-java -cp $JAVA_HOME/lib/tools.jar:repairnator-pipeline.jar -Dlogback.configurationFile=/root/logback.xml fr.inria.spirals.repairnator.pipeline.Launcher -d --runId $LOCAL_RUN_ID -o $LOCAL_OUTPUT --ghOauth $LOCAL_GITHUB_OAUTH --repairTools $REPAIR_TOOLS $args
+if [ "$REPAIR_MODE" == "FAULT_LOCALIZATION" ]; then
+    echo "java -cp $JAVA_HOME/lib/tools.jar:repairnator-pipeline.jar -Dlogback.configurationFile=/root/logback.xml fr.inria.spirals.repairnator.pipeline.Launcher -d --runId $LOCAL_RUN_ID -o $LOCAL_OUTPUT --ghOauth $LOCAL_GITHUB_OAUTH --faultLocalization --flacocoThreshold $FLACOCO_THRESHOLD --flacocoTopK $FLACOCO_TOP_K --flacocoResultsRepository $FLACOCO_RESULTS_REPOSITORY --ghOauth $GITHUB_TOKEN $args
+"
+    java -cp $JAVA_HOME/lib/tools.jar:repairnator-pipeline.jar -Dlogback.configurationFile=/root/logback.xml fr.inria.spirals.repairnator.pipeline.Launcher -d --runId $LOCAL_RUN_ID -o $LOCAL_OUTPUT --ghOauth $LOCAL_GITHUB_OAUTH --faultLocalization --flacocoThreshold $FLACOCO_THRESHOLD --flacocoResultsRepository $FLACOCO_RESULTS_REPOSITORY --ghOauth $GITHUB_TOKEN $args
+else
+    java -cp $JAVA_HOME/lib/tools.jar:repairnator-pipeline.jar -Dlogback.configurationFile=/root/logback.xml fr.inria.spirals.repairnator.pipeline.Launcher -d --runId $LOCAL_RUN_ID -o $LOCAL_OUTPUT --ghOauth $LOCAL_GITHUB_OAUTH --repairTools $REPAIR_TOOLS $args
+fi

src/repairnator-core/src/main/java/fr/inria/spirals/repairnator/GitRepositoryLauncherUtils.java

@@ -24,6 +24,8 @@ public static void registerGitArgs(JSAP jsap) throws JSAPException {
         jsap.registerParameter(GitRepositoryLauncherUtils.defineArgGitRepositoryIdCommit());
         // --gitRepoFirstCommit
         jsap.registerParameter(GitRepositoryLauncherUtils.defineArgGitRepositoryFirstCommit());
+        // --gitRepoPullRequest
+        jsap.registerParameter(GitRepositoryLauncherUtils.defineArgGitRepositoryPullRequest());
     }
 
     public static void initGitConfig(RepairnatorConfig config, JSAPResult arguments, JSAP jsap) {
@@ -44,6 +46,7 @@ public static void initGitConfig(RepairnatorConfig config, JSAPResult arguments,
         config.setGitRepositoryUrl(GitRepositoryLauncherUtils.getArgGitRepositoryUrl(arguments));
         config.setGitRepositoryBranch(GitRepositoryLauncherUtils.getArgGitRepositoryBranch(arguments));
         config.setGitRepositoryIdCommit(GitRepositoryLauncherUtils.getArgGitRepositoryIdCommit(arguments));
+        config.setGitRepositoryPullRequest(GitRepositoryLauncherUtils.getArgGitRepositoryPullRequest(arguments));
     }
 
     public static FlaggedOption defineArgGitRepositoryUrl() {
@@ -94,6 +97,19 @@ public static boolean getArgGitRepositoryFirstCommit(JSAPResult arguments) {
         return arguments.getBoolean("gitRepositoryFirstCommit");
     }
 
+    public static FlaggedOption defineArgGitRepositoryPullRequest() {
+        FlaggedOption opt = new FlaggedOption("gitRepositoryPullRequest");
+        opt.setLongFlag("gitrepopullrequest");
+        opt.setStringParser(JSAP.INTEGER_PARSER);
+        opt.setDefault("-1");
+        opt.setHelp("Specify the pull request to analyze  (only in FAULT_LOCALIZATION mode).");
+        return opt;
+    }
+
+    public static Integer getArgGitRepositoryPullRequest(JSAPResult arguments) {
+        return arguments.getInt("gitRepositoryPullRequest");
+    }
+
     public static List<SerializerEngine> initFileSerializerEngines(Logger logger) {
         List<SerializerEngine> fileSerializerEngines = new ArrayList<>();
         RepairnatorConfig config = RepairnatorConfig.getInstance();

src/repairnator-core/src/main/java/fr/inria/spirals/repairnator/GithubInputBuild.java

@@ -8,6 +8,7 @@ public class GithubInputBuild implements InputBuild {
     private String url;
     private String branch;
     private String sha;
+    private Integer pr;
 
     public GithubInputBuild(String url, String branch, String sha)
     {
@@ -16,6 +17,11 @@ public GithubInputBuild(String url, String branch, String sha)
         this.branch = branch;
     }
 
+    public GithubInputBuild(String url, String branch, String sha, Integer pr) {
+        this(url, branch, sha);
+        this.pr = pr;
+    }
+
     public String getSha() {
         return sha;
     }
@@ -32,18 +38,23 @@ public String getSlug() {
         return url.split("https://github.com/",2)[1];
     }
 
+    public Integer getPr() {
+        return pr;
+    }
+
     @Override
     public List<String> getEnvVariables() {
         List<String> r = new ArrayList<>();
 
         r.add("GITHUB_URL=" + url);
         r.add("GITHUB_BRANCH=" + sha);
         r.add("GITHUB_SHA=" + sha);
+        r.add("GITHUB_PR=" + pr);
         return r;
     }
 
     @Override
     public String toString() {
-        return url + "-" + branch + "-" + sha;
+        return url + "-" + branch + "-" + sha + "-" + pr;
     }
 }

src/repairnator-core/src/main/java/fr/inria/spirals/repairnator/LauncherUtils.java

@@ -121,6 +121,10 @@ public static void registerCommonArgs(JSAP jsap) throws JSAPException {
 
         // --flacocoTreshold
         jsap.registerParameter(LauncherUtils.defineArgFlacocoThreshold());
+        // --flacocoTopK
+        jsap.registerParameter(LauncherUtils.defineArgFlacocoTopK());
+        // --flacocoResultsRepository
+        jsap.registerParameter(LauncherUtils.defineArgFlacocoResultsRepository());
 
         // --bears
         jsap.registerParameter(LauncherUtils.defineArgBearsMode());
@@ -239,6 +243,8 @@ public static void initCommonConfig(RepairnatorConfig config, JSAPResult argumen
         config.setNPERepairStrategy(LauncherUtils.getArgNPERepairStrategy(arguments));
 
         config.setFlacocoThreshold(LauncherUtils.getArgFlacocoThreshold(arguments));
+        config.setFlacocoTopK(LauncherUtils.getArgFlacocoTopK(arguments));
+        config.setFlacocoResultsRepository(LauncherUtils.getArgFlacocoResultsRepository(arguments));
 
         config.setPatchClassificationMode(LauncherUtils.getArgPatchClassificationMode(arguments));
         config.setPatchClassification(LauncherUtils.getArgPatchClassification(arguments));
@@ -940,7 +946,7 @@ public static FlaggedOption defineArgFlacocoThreshold() {
         FlaggedOption opt = new FlaggedOption("flacocoThreshold");
         opt.setLongFlag("flacocoThreshold");
         opt.setStringParser(JSAP.DOUBLE_PARSER);
-        opt.setDefault("0.5");
+        opt.setDefault("0.1");
         opt.setHelp("Threshold for flacoco fault localization");
         return opt;
     }
@@ -949,6 +955,32 @@ public static Double getArgFlacocoThreshold(JSAPResult arguments) {
         return arguments.getDouble("flacocoThreshold");
     }
 
+    public static FlaggedOption defineArgFlacocoTopK() {
+        FlaggedOption opt = new FlaggedOption("flacocoTopK");
+        opt.setLongFlag("flacocoTopK");
+        opt.setStringParser(JSAP.INTEGER_PARSER);
+        opt.setDefault("5");
+        opt.setHelp("Number of results to be pushed by flacocobot");
+        return opt;
+    }
+
+    public static Integer getArgFlacocoTopK(JSAPResult arguments) {
+        return arguments.getInt("flacocoTopK");
+    }
+
+    private static Parameter defineArgFlacocoResultsRepository() {
+        FlaggedOption opt = new FlaggedOption("flacocoResultsRepository");
+        opt.setLongFlag("flacocoResultsRepository");
+        opt.setStringParser(JSAP.STRING_PARSER);
+        opt.setDefault("");
+        opt.setHelp("Repository where to save the results of flacocobot");
+        return opt;
+    }
+
+    private static String getArgFlacocoResultsRepository(JSAPResult arguments) {
+        return arguments.getString("flacocoResultsRepository");
+    }
+
     public static FlaggedOption defineArgODSPath() {
         FlaggedOption opt = new FlaggedOption("odsPath");
         opt.setLongFlag("odsPath");

src/repairnator-core/src/main/java/fr/inria/spirals/repairnator/config/RepairnatorConfig.java

@@ -113,6 +113,7 @@ public enum PATCH_FILTERING_MODE {
     private String gitRepositoryBranch;
     private String gitRepositoryIdCommit;
     private boolean gitRepositoryFirstCommit;
+    private Integer gitRepositoryPullRequest;
 
     private String[] sonarRules;
     private boolean isStaticAnalysis;
@@ -124,6 +125,8 @@ public enum PATCH_FILTERING_MODE {
     private String npeRepairStrategy;
 
     private Double flacocoThreshold;
+    private Integer flacocoTopK;
+    private String flacocoResultsRepository;
 
     private PATCH_CLASSIFICATION_MODE patchClassificationMode;
     private PATCH_FILTERING_MODE patchFilteringMode;
@@ -141,6 +144,7 @@ public enum PATCH_FILTERING_MODE {
     // Realtime
     private File whiteList;
     private File blackList;
+    private File projectsToScan;
     private int jobSleepTime;
     private int buildSleepTime;
     private int maxInspectedBuilds;
@@ -551,6 +555,14 @@ public void setBlackList(File blackList) {
         this.blackList = blackList;
     }
 
+    public File getProjectsToScan() {
+        return projectsToScan;
+    }
+
+    public void setProjectsToScan(File projectsToScan) {
+        this.projectsToScan = projectsToScan;
+    }
+
     public int getJobSleepTime() {
         return jobSleepTime;
     }
@@ -783,6 +795,22 @@ public void setFlacocoThreshold(Double flacocoThreshold) {
         this.flacocoThreshold = flacocoThreshold;
     }
 
+    public Integer getFlacocoTopK() {
+        return flacocoTopK;
+    }
+
+    public void setFlacocoTopK(Integer flacocoTopK) {
+        this.flacocoTopK = flacocoTopK;
+    }
+
+    public String getFlacocoResultsRepository() {
+        return flacocoResultsRepository;
+    }
+
+    public void setFlacocoResultsRepository(String flacocoResultsRepository) {
+        this.flacocoResultsRepository = flacocoResultsRepository;
+    }
+
     @Override
     public String toString() {
         String ghToken = this.getGithubToken();
@@ -856,6 +884,8 @@ public String toString() {
                 ", jTravisEndpoint=" + jTravisEndpoint +
                 ", travisToken=" + travisToken +
                 ", flacocoThreshold=" + flacocoThreshold +
+                ", flacocoTopK=" + flacocoTopK +
+                ", flacocoResultsRepository=" + flacocoResultsRepository +
                 '}';
     }
 
@@ -931,7 +961,15 @@ public void setGitRepositoryFirstCommit(boolean gitRepositoryFirstCommit) {
 		this.gitRepositoryFirstCommit = gitRepositoryFirstCommit;
 	}
 
-	public String getGitRepositoryId() {
+    public Integer getGitRepositoryPullRequest() {
+        return gitRepositoryPullRequest;
+    }
+
+    public void setGitRepositoryPullRequest(Integer gitRepositoryPullRequest) {
+        this.gitRepositoryPullRequest = gitRepositoryPullRequest;
+    }
+
+    public String getGitRepositoryId() {
 		return getGitRepositoryUrl().split("https://github.com/",2)[1].replace(".git","").replace("/", "-") + "-" + (getGitRepositoryBranch() != null ? getGitRepositoryBranch() : "master") +
 				(getGitRepositoryIdCommit() != null ? "-" + getGitRepositoryIdCommit() : "") +
 				(isGitRepositoryFirstCommit() ? "-firstCommit" : "");