[Replication package] Tracto - A Neuro-Symbolic Approach to Deriving Concrete Test Oracles

This repository contains the replication package of Tracto, a neuro-symbolic approach to generate concrete test oracles from Java projects.

Tracto stands for Transformer-based Concrete Test Oracle generation and reminds the sound of the italian word tratto that means line or way, which illustrates the way in which Tracto generates oracles, as if they were a line that is continuously drawn, little by little (token by token). Tracto generates oracles token by token, in a grammar-directed way, but supported by a neural module to guide the search of oracles toward optimal solutions.

Directory structure

1. ./scripts/miner contains the scripts to mine the test cases after a given date, to perform experiments on unseen data, and avoid data leakeage.
2. ./scripts/neural-module contains the scripts related to the neural module (fine-tuning and inference).
3. ./scripts/prompt contains the scripts to generate the prompts for the neural module (both for fine-tuning and inference)
4. ./scripts/symbolic-module contains the code related to the symbolic module.
5. rqs contains the code and the data to replicate the experiments described in the related paper.

Setup

To set up and run the fine-tuned model and the vanilla models used in the experiments with Ollama, the following requirements must be satisfied:

• Python ≥ 3.12
• Pip

The experiments were performed on a GPU Nvidia A-100 with 80GB of memory (vRAM) and 147GB of RAM.

1. Environment

In order to set up the environment is not strictly necessary, but recommended to create a venv or conda environment. In the following section we will provide the general information to configure the environment.

1.A.1 Create and activate venv environment

Follow the instructions to properly create and activate the environment

1. Open a terminal and move to the root directory of the project.
2. Create the venv environment with Python ≥ 3.10:

   python3 -m venv .venv

   The command will generate a venv environment within .venv.
3. Activate the venv environment:

   source .venv/bin/activate

4. Upgrade pip and setuptools to the latest version:

   pip install --upgrade pip setuptools

1.A.2 Install the requirements

Install the dependencies within the active venv environment, using the following command:

python3 -m pip install --upgrade -r requirements.txt --target=[path_to_repo]/.venv/lib/[python_version]/site-packages

where:

• path_to_repo must be substituted the absolute path to the replication package repository
• python_version must be substituted with the version of Python used in the environment (e.g., python3.12)

1.B.1 Install Conda

The reader can find all the instructions to properly install and setup conda, on the official user guide.

We provide an example on how to install conda on Linux operating systems. The installation is relatively similar for macOS and Windows systems.

1. Open a terminal and type the following command to download the installer:

   wget -P [path_to_destination_directory] [conda_download_http_link]

   • path_to_destination_directory - represents the destination path where the installer will be downloaded.
   • conda_download_http_link - is the link of the installer. You can choose the installer here, copying the corresponding url.

2. Execute the installer:

   bash [fullname_of_the_installer]

   • fullname_of_the_installer - is the whole name of the downloaded installer (it is a .sh file, like for example Miniconda3-py39_23.3.1-0-Linux-aarch64.sh)

3. Close and re-open the terminal at the end of the installation. Check conda has been successfully installated typing the command:

   conda --version

1.B.2 Create and activate conda environment

4. Create a new conda environment named llm_prompts with Python version ≥ 3.10 and pip.

    conda create --name tracto python=[python_version] pip

   where:

   • python_version must be substituted with the version of Python used in the environment (e.g., 3.12)

5. Activate the conda environment

   conda activate llm_prompts

1.B.3 Install the requirements

6. Move to the root folder of the repository

7. Install all the required dependencies

   pip install -r requirements.txt

8. Bash requirements

In order to run the bash scripts, the following requirements must be satisfied:

• jq
• zip
• unzip
• git