FlyCatcher: Neural Inference of Runtime Checkers from Tests

This repository contains the implementation of FlyCatcher and supplementary material for the paper "FlyCatcher: Neural Inference of Runtime Checkers from Tests"

Execution Environment

Follow the requirements and instalation instructions to set up the execution environment in ENVIRONMENT.md.

Replication Guide

To reproduce the results from the paper, follow these instructions. The results of the following instructions are provided in the Available Data section bellow, i.e., you can also inspect them there to skip some of the below steps.

Get Target Tests and Related Static Information:

1. Install CodeQL for VS Code: https://docs.github.com/en/code-security/codeql-for-vs-code/getting-started-with-codeql-for-vs-code/installing-codeql-for-vs-code

2. Manage the CodeQL database to analyze: Create the database using CodeQL CLI (This might be the way to go in case of specific project versions):

   1. Install CodeQL CLI: https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli

   2. Get project:

   3. cd data/projects

   4. git clone *project* (e.g. git clone https://github.com/apache/zookeeper)

   5. cd *project* & git checkout *version* (e.g. git checkout tags/release-3.4.11)

   6. codeql database create *project*-db --language=java --source-root=*project root* (codeql database create zk-db --language=java --build-mode=none --source-root=.)

3. Run query: Select one of the queries in the ./codeql-custom-queries-java folder and run it. Then, save the output as CSV file to ./data/static/project/raw/id.csv. Where id is the name of the .ql file without the get-, e.g. tests-and-methods-called-by-them.

4. Join the query results:

   python ProcessStaticData.py --files_root=*path* --project_root=*path* (python ProcessStaticData.py --files_root=data/static/zookeeper/raw --project_root=data/projects/zookeeper/)

5. Build the target system:

   cd data/projects/hbase & mvn package -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.2
   

6. Copy and fix dependencies:

   mvn dependency:copy-dependencies -DoutputDirectory=./lib -DincludeScope=runtime,compile,test
   
   find . -type f -name "asm*.jar" -delete
   find . -type f -name "jopt-simple*.jar" -delete
   

7. Run each test class and add the failing ones to ./GetTargetTests_system.py

8. Randomly select 100 targets:

   python GetTargetTests_system.py --input tests_and_their_info.csv (python GetTargetTests_hb.py --input ./data/static/hbase/tests_and_their_info.csv)
   

Generate and Validate Checkers:

python Main.py --input=./data/static/cassandra/tests_and_their_info.csv --target=./data/target_tests_cs.csv --project_root=./data/projects/cassandra --dependencies_cp ./data/projects/cassandra/build/classes/main:./data/projects/cassandra/build/classes/thrift:./data/projects/cassandra/build/test/classes:./data/projects/cassandra/lib/*:./data/projects/cassandra/build/lib/jars/*

Cross-Validate Checkers:

python ./evaluation/CrossValidation.py --results ./data/results_claude_cs.csv --instrumented_classes ./data/output/instrumented_classes_all/ --target ./data/target_tests_cs.csv --dependencies_cp ./data/projects/cassandra/build/classes/main:./data/projects/cassandra/build/classes/thrift:./data/projects/cassandra/build/test/classes:./data/projects/cassandra/lib/*:./data/projects/cassandra/build/lib/jars/*

Mutation Analysis

python ./evaluation/MutationAnalysis.py --input=./data/static/zookeeper/tests_and_their_info.csv --results=./data/results_claude_zk.csv --src_dir=./data/projects/zookeeper/src --src_dir_byte=./data/projects/bytecode/zookeeper-3.4.11 --tests_dir_byte=./data/projects/bytecode/zookeeper-3.4.11-tests --dependencies_cp dependencies

Study on Feedback contributions

python ./evaluation/GetFeedbackContributions.py

Costs of the Approach

1. Measure the execution time with and without the checkers activated:

python ./evaluation/MeasureExecutionTime.py

2. Summarize the results:

python ./evaluation/SummarizeExecutionTime.py

Available Data

The data from our experiments can be found in the following folders in the artifact:

1. The target tests data can be found at:

./data/target_tests_*.csv

./data/static

2. The prompts, model responses, instrumented classes, validation and cross-validation results for each target test, checker and model are available at:

./data/output/

3. The mutation analysis is available at:

./data/mutation_analysis_zk

4. The overhead analysis is available at:

./data/overhead_analysis