CodeQL Baseline for CWE-Bench-Java

Dockerized CodeQL runner for evaluating CodeQL on CWE-Bench-Java cases.

This repository is self-contained for running CodeQL on CWE-Bench-Java metadata: it includes the runner, Docker build files, IRIS build metadata, and data/cwe_bench_java.csv. Vulnerable repositories are cloned on demand into the repo cache directory.

Environment

The Docker image pins:

• CodeQL bundle v2.26.1
• Eclipse Temurin JDK 8
• Apache Maven 3.2.1, 3.5.0, and 3.9.8
• Python 3

The default platform is linux/amd64.

Run inside VDAgent-experiment

From the VDAgent-experiment project root:

bash baselines/codeql/run_in_docker.sh \
  --use-iris-build-metadata \
  --cve-id CVE-2020-2261 \
  --rerun

Run all cases:

bash baselines/codeql/run_in_docker.sh \
  --use-iris-build-metadata

Run as a Standalone Repository

Clone this repository on the server:

git clone https://github.com/OasisPeng/codeql.git
cd codeql

Run one case:

bash run_in_docker.sh \
  --use-iris-build-metadata \
  --cve-id CVE-2020-2261 \
  --rerun

Run all cases:

bash run_in_docker.sh \
  --use-iris-build-metadata

By default, repositories are cloned on demand into repos/, and outputs are written to results/codeql/.

You may also pass paths explicitly:

bash run_in_docker.sh \
  --dataset /path/to/cwe_bench_java.csv \
  --repo-dir /path/to/repo-cache \
  --output-dir /path/to/codeql-results \
  --use-iris-build-metadata \
  --cve-id CVE-2020-2261 \
  --rerun

Alternatively set VDAGENT_PROJECT_ROOT to the VDAgent-experiment project root:

VDAGENT_PROJECT_ROOT=/path/to/VDAgent-experiment \
bash run_in_docker.sh --use-iris-build-metadata

Outputs

results/codeql/raw/<CVE>_<CWE>/db/
results/codeql/raw/<CVE>_<CWE>/results.sarif
results/codeql/raw/<CVE>_<CWE>/metadata.json
results/codeql/summary.csv
results/codeql/eval_summary.json
results/codeql/run_metadata.json

When run inside VDAgent-experiment, outputs default to baselines/results/codeql.

Detection Rule

A CVE is counted as detected when at least one CodeQL SARIF alert overlaps with the CWE-Bench-Java ground-truth location by file path and line range.

Time Accounting

elapsed_seconds measures only the CodeQL phase after repository preparation. It excludes repository clone/fetch/checkout time.

Notes

There is no automatic fallback from autobuild to build-mode=none. If a case should be analyzed without building, pass it explicitly:

bash run_in_docker.sh --build-mode none --cve-id CVE-2020-2261

For full 120-case experiments, prefer a native x86_64 Linux server. Running the linux/amd64 image through emulation on an ARM Mac can be much slower.