Test-DB: Totally Experimental SQLite Tester

Part 1 of the Project from the Automated Software Testing course @ ETH Zurich.

Author	Links
Lennart K. M. Schulz	GitHub, LinkedIn
Dovydas Vadišius	GitHub, LinkedIn

Report, Project Description

Fuzzer UI and code coverage metrics after 2 hours of running.

Installation

The tool can either be installed locally:

pip install .

or run as a docker container:

docker build . --platform linux/amd64 -t test-db:latest
docker run test-db:latest <command>

The main entrypoint (test-db) is on PATH in the container.

Alternatively, the docker-compose file can be used to run the container in the background with the current directory mounted at ~/repo and then get an interactive shell into the container:

docker compose up -d
docker exec -it <container_name> /bin/bash

For development purposes it may help to use an editable pip installation (either locally or in the docker container):

pip install --editable .

This installation will always reflect the changes made to the source files without re-installation.

Requirements

Note that the docker image already has all requirements satisfied.

For a local installation, however it is important that python3.12 is installed and used to run the fuzzer. All dependencies are then installed automatically when doing pip install ..

To run the fuzzer, two SQLite binaries are needed; one as oracle, and one as engine under test.

Fuzzer

The fuzzer can be started as follows:

test-db [Flags] <engine>

where engine should be a valid SQLite binary (either as relative/absolute file path or as command found on PATH).

The complete flags and usage options are shown below.

usage: test-db [-h] [--oracle ORACLE] [-t TIME] [-p PROCESSES] [-o OUT] [-a] [-r THRESH_GEN_RST] [-w WIDE_CHANCE] [-v] [-l LEVEL] engine

positional arguments:
  engine                (path of) db engine to be fuzzed

options:
  -h, --help            show the help message and exit
  --oracle ORACLE       (path of) db engine to use as oracle (default: sqlite3)
  -t TIME, --time TIME  runtime of the fuzzer (format: (days:)hours:minutes, default: infinite)
  -p PROCESSES, --processes PROCESSES
                        number of concurrent workers (default: number of logical cores / 2)
  -o OUT, --out OUT     directory to use for outputs (default: './test_db-out')
  -a, --abandon-schemas
                        abandon schema after the first logic bug
  -r THRESH_GEN_RST, --thresh-gen-rst THRESH_GEN_RST
                        number of errors or timeouts after which to reset the generator parameters (default: 25, infinite: 0)
  -w WIDE_CHANCE, --wide WIDE_CHANCE  
                        chance of using wider 'antrl' grammar instead of native one (default: 0.05)
  -v, --verbose         enable verbose output
  -l LOG_LEVEL, --log-level LOG_LEVEL
                        explicitly set the logging level

Replay Utility

Test-DB includes a replay utility that can be used to replay bug reports (e.g., query minimization purposes).

The tool can be invoked as any python module, i.e.,

python3 -m test_db.replay

The complete flags and usage options are shown below.

usage: python3 -m test_db.replay [-h] [-q QUERY_FILE] [-b BASE_PATH] [-e ENGINE] [--oracle ORACLE] bug_id

positional arguments:
  bug_id                ID of original bug report

options:
  -h, --help            show this help message and exit
  -q QUERY_FILE, --query-file QUERY_FILE
                        path of file with query, if left empty, rerun with original query
  -b BASE_PATH, --base-path BASE_PATH
                        path of directory with bug reports (default: './test_db-out/bug-reports')
  -e ENGINE, --engine ENGINE
                        (path of) db engine (default: use engine from version.txt)
  --oracle ORACLE       (path of) db engine to use as oracle (default: 'sqlite3')