ParSEval: Plan-aware Test Database Generation for SQL Equivalence Evaluation

ParSEval considers the specific behaviors of each query operator and covers all possible execution branches of the logical query plan by adapting the notion of branch coverage to query plans.

File Structure

The repo contains following supplemental materials:

• source code of ParSEval
• Source code of query parser

├── src # Source code of ParSEval
├── requirements.txt # pip requirements
└── README.md

Get started

Install the Query Parser

Please download and set up the query parser from the repository.

Set Up the Python Environment

1. Please use conda or venv to create a virtual environment. Run following command to install requirements.

# Example with venv
python -m venv venv
source venv/bin/activate
# Or with conda
conda create -n parseval-dev python=3.8
conda activate parseval-dev

2. Install the required dependencies:

pip install -r requirements.txt

Usage

Normally, one invoke the tool as

python main.py --schema SCHEMA --dialect sqlite --gold SQL1 --offline

to generate test database instances for input query SQL1.

To test the equivalence of two queries:

python main.py --schema SCHEMA --dialect sqlite --gold SQL1 --pred SQL2

One can enhance the readability of generated data for common column types by customizing the data generation strategy in the register_default_generators function.

# Integer generator
def int_generator(existing_values: Optional[Set[Any]] = None, is_unique: bool = False) -> int:
    """
    Generate a random integer value.        
    Args:
        existing_values: Set of existing values to avoid duplicates if is_unique is True
        is_unique: Whether the value should be unique
        
    Returns:
        int: A random integer value
    """
    value = random.randint(1, 100)
    if is_unique and existing_values:
        while value in existing_values:
            value = random.randint(1, 100)
    return value
from .registry import ValueGeneratorRegistry
ValueGeneratorRegistry.register('int', int_generator)

Experiment Setup

• Install Docker
• Dataset
  • Download Leetcode/Literature/Bird/Spider datasets here.
  • Could also download official database instances for bird and spider.

Running Experiments

Commands needed can be found in the tests folder.

Planed features

• NULL-related constraints