What is REQBench?

REQBench is a large-scale benchmark with 2,095 test cases, each labeled with compatibility information. This large-scale dataset serves as a benchmark for evaluating compatible requirements inference approaches of TPL upgrades in Python projects.

Label Folder Explanation

Each .json file in the label directory represents a different scenario used to determine compatibility in TPL upgrades:

File Name	Meaning When false
pip.json	Pip failed to resolve dependencies
module.json	Pip unresolved; issue due to module
apiname.json	Pip unresolved; issue due to API name
apiparameter.json	Pip unresolved; issue due to API parameter
apibody.json	Pip unresolved; issue due to API body
project_tpl.json	Pip unresolved; issue at the project-TPL level
tpl_tpl.json	Pip unresolved; issue at the TPL-TPL level

Projects

The source code of each Python project can be accessed at Zenodo.

Example Directory Layout

./
├── 3d-ken-burns.tar.gz                            # Project source code
...

🛠️ Extracting

To extract the project, run:

cd /home/usr/
mkdir projects
tar -xzvf projectName.tar.gz -C projects

How Does PCREQ Use It?

Each test case in REQBench can be regarded as an independent test project. To use PCREQ to infer compatible requirements in REQBench.

Write Configuration Files Manually

{
    "projPath": "/home/usr/projects/project",
    "requirementsPath": "/home/usr/requirements/project/requirements.txt",
    "targetLibrary": "pillow",
    "startVersion": "6.2.0",
    "targetVersion": "7.0.0",
    "pythonVersion": "3.7",
    "knowledgePath": "/home/usr/knowledge/"
}

Then, copy the configuration file to PCREQ/configure and run the command as follows:

python main.py --config config.json

All configuration files are located in the configure folder.