A Python benchmark for evaluating code generated by large language models, with audited ground truth and a held-out test split.
CodeAssay grades a generated program on functional correctness, measured only on tests the model was never shown, and reports selected code properties for the programs that pass. A strict gate credits those properties only when a program passes every hidden test.
No composite score is formed. Correctness and each property are reported separately, so no weighting is imposed on the reader.
This repository is the replication package for the CodeAssay paper. It contains the benchmark, the harness, every generated program, and the analysis behind the reported numbers. See REPRODUCIBILITY.md for the execution environment, tool versions, and a file-by-file inventory.
Formerly named VibeBench and hosted under a personal account. GitHub redirects the earlier
shahbazsiddeeq/vibebenchandshahbazsiddeeq/CodeAssayURLs here, but please cite the current one.
tasks/python/ 185 tasks: task.yaml, reference solution,
tests_public/, tests_hidden/, tests/ (their union)
runner/ Grading engine
scripts/ Generation drivers, split tooling, analysis
configs/ The 14 run configurations and metric definitions
reports/ Published validation and statistical outputs
.agent_runs/r2-*/ Every generated program, per-call logs, run settings
REPRODUCIBILITY.md Environment, tool versions, artifact inventory
185 Python tasks across ten categories informed by SWEBOK v4 knowledge areas and the domain structure of BigCodeBench:
| Code | Category | Tasks |
|---|---|---|
| ADS | Algorithms & Data Structures | 30 |
| SPT | String & Text Processing | 26 |
| DPS | Data Processing & Statistics | 21 |
| FIO | File I/O & System | 18 |
| WNT | Web & Networking | 16 |
| CCA | Concurrency & Async | 16 |
| CAT | Crypto, Auth & Tokens | 15 |
| DBS | Database & Storage | 15 |
| TCQ | Testing & Code Quality | 15 |
| ODP | OOP & Design Patterns | 13 |
Every task carries a description, a signature, an author-written reference
solution, and a pytest suite, so a generated solution replaces the reference
without project-level scaffolding.
The public/hidden split
Each task's test functions are partitioned into a public subset and a disjoint hidden subset. Only the public tests reach a model, as an excerpt in the prompt and as repair feedback. No hidden test is shown, executed, or summarized during generation or repair.
The partition is deterministic and independent of test content: top-level
test_* functions are taken in source order and assigned by alternating
position, so earlier and later tests fall on both sides. Where the count is odd
the extra test goes to the hidden subset.
The 1,697 test functions divide into 799 public and 898 hidden, and every
task keeps at least two on each side. The per-task manifest is
tasks/python/test_split_manifest.json; tests/ in each task directory is the
union of the two and is what mutation analysis runs against.
Reference solutions average 20.3 source lines. Only one task depends on a third-party package, so evaluation needs no network access.
Tasks, reference solutions, and tests were authored for this benchmark rather
than adapted from an existing suite. A similarity audit against all 1,138
HumanEval and MBPP tasks found a maximum of 0.48 for descriptions and
0.64 for reference solutions, below the prespecified threshold of 0.70
(reports/leakage_similarity.csv). That audit compares surface form only. It
shows no evidence of derivation from those two benchmarks; it cannot rule out
semantic overlap, overlap with other sources, or presence in any model's
training data.
The benchmark was audited before any model was scored. Every task's reference was attacked, descriptions were checked against test expectations, and suites were strengthened until they reject deliberately incorrect solutions.
Mutation analysis validates the test suites rather than scoring models.
mutmut generates mutants from each reference solution and runs that task's
suite against every one:
| Suite | Mutants | Killed | Score |
|---|---|---|---|
| Full | 4,008 | 3,311 | 0.826 |
| Hidden only | 4,008 | 2,999 | 0.748 |
The hidden subset, which is what grading uses, retains 90.6% of the detections the full suites achieve. Equivalent mutants were not identified, so both scores are lower bounds.
Seven models under a standard and a security-focused prompt, giving 14 configurations. Each generates one program per task, with at most one repair driven by the public tests. Grading uses the hidden tests only.
Correctness, from reports/r2_results_summary.csv (185 tasks each):
| Configuration | Correct | % | Configuration | Correct | % |
|---|---|---|---|---|---|
| gpt56sol-std | 183 | 98.9 | haiku45-sec | 175 | 94.6 |
| sonnet5-std | 183 | 98.9 | gemini25-std | 169 | 91.4 |
| gpt56sol-sec | 182 | 98.4 | gemini25-sec | 169 | 91.4 |
| sonnet5-sec | 181 | 97.8 | gpt4o-sec | 166 | 89.7 |
| haiku45-std | 179 | 96.8 | gpt4o-std | 161 | 87.0 |
| sonnet45-sec | 177 | 95.7 | gpt4omini-std | 143 | 77.3 |
| sonnet45-std | 176 | 95.1 | gpt4omini-sec | 139 | 75.1 |
Models are proprietary and API-served. Because closed models can change behind a
fixed identifier, regenerating outputs will not reproduce these numbers. The
stored programs under .agent_runs/r2-*/ allow the analysis to be reproduced
exactly.
This is the main reason the generated programs are released. Nothing here calls a provider.
python3.12 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python3.12 scripts/verify_and_repair_audit.py # regrade every stored program
python3.12 scripts/gen_r2_results_summary.py # -> reports/r2_results_summary.csv
python3.12 scripts/rev_stats_v2.py # correctness and quality statistics
python3.12 scripts/per_category_correctness.py # per-category breakdown
python3.12 scripts/r2_repair_audit.py # repair and token accountingGrading pins PYTHONHASHSEED=0, so a solution that iterates a set or dict emits
in a fixed order and regrading is deterministic.
Suite validation and the contamination audit:
python3.12 scripts/mutation_validate.py --tests tests # full suites
python3.12 scripts/mutation_validate.py --tests tests_hidden # hidden only
python3.12 scripts/run_semgrep_scan.py
python3.12 scripts/leakage_check.pyRebuilding the public/hidden split from scratch, and checking the two subsets share no test function:
python3.12 scripts/make_test_split.py
python3.12 scripts/verify_split.pycp .env.example .env # add your OpenAI / Anthropic / OpenRouter keys
set -a; . ./.env; set +a
python3.12 scripts/run_agents.py --config configs/r2_sonnet5_std.yamlOutput goes to .agent_runs/<config>/. The 14 configurations behind the
published results are configs/r2_*.yaml.
| Quantity | Tool | Role |
|---|---|---|
| Correctness | pytest, hidden | Primary metric; all hidden tests must pass |
| Style | flake8 | Violations per 100 logical lines |
| Cyclomatic complexity | radon | Mean and maximum per solution |
| Static-analysis findings | bandit, semgrep | Counts by severity, reported descriptively |
| Length | radon | Logical lines, reported as context |
| Mutation score | mutmut | Validates the suites, not the models |
Each property is analyzed only for solutions that pass. The static-analysis counts detect known insecure constructions rather than semantic vulnerabilities, and are not a measure of program security. Analyzer versions determine the numeric values; they are pinned in REPRODUCIBILITY.md.
The runner also emits a pip-audit dependency check. It is not among the
reported measures.
Legacy note.
configs/metrics.v1*.jsondefine a weighted composite and stamp identifiers beginningVibeBench-v1into results they produce. They are kept so earlier runs stay reproducible. The composite is not used in the published results and is not a benchmark score.
- Pre-revision results (May 2026). An earlier evaluation covering 120 tasks and five models used a composite score and graded on tests the model had been shown. Those results are invalid and are not published.
- Manuscript sources and figure-generation code. Out of scope for a
replication package. Every number a figure displays is in
reports/.
MIT. See LICENSE. This covers the tasks and their tests as well as the code.
scripts/leakage_check.py downloads HumanEval and MBPP at run time for the
similarity audit; those benchmarks carry their own licences and are not
redistributed here.
See CITATION.cff.