Auditing the rigor of RTL design benchmarks via mutation testing.
A testbench that never fails isn't proof of a correct design — it may just never
stimulate what's actually broken.
Benchmarks for LLM-generated RTL treat their own testbenches as ground truth, and almost none check whether that trust is earned. GateTruth is a mutation-testing engine: inject deterministic, seeded semantic faults into a reference design and measure what fraction the testbench actually catches. A testbench that lets a broken design pass is a testbench that will also let a broken model-generated design pass — and no amount of pass@k tells you which kind you have.
flowchart LR
R[Reference design] -- seeded, deterministic faults --> M[Mutants]
M --> T{{Testbench}}
T -- caught --> K["✅ Killed"]
T -- missed --> S["❌ Survived<br/>= a fault the testbench<br/>can't actually see"]
K & S --> Score["Kill rate<br/>= rigor, measured"]
Score -.->|"same engine,<br/>unmodified"| Ext["Applied to RTLLM v2.0<br/>(and our own suite first)"]
We certify the methodology against our own open reference suite first — 60 specification-to-RTL generation tasks plus 8 agentic-repair tasks, each carried through a real, pinned open-source ASIC flow (synthesis + static timing/power on the sky130 technology) with correctness enforced as a hard gate. 46 of the 60 Track A testbenches clear a ≥95% mutation-kill floor, counting only behavioral simulation failures as kills; the other 14 don't, and we report that as a finding rather than an omission — see the paper for exactly which ones and why. Only then do we point the same engine, unmodified, at an external RTL benchmark the field already relies on. The paper has the full audit methodology, the reference-suite design, and the resulting measurements.
Seven models, evaluated under the official pinned flow at a common 16,384-token output cap (an earlier 4096-token run materially reordered this table — see the paper for why the cap itself turned out to be an experimental variable). The human reference scores 66.67 by construction; every model trails it.
| Rank | Model | Score / 100 | Pass@1 |
|---|---|---|---|
| 🥇 | GPT-5 | 48.83 | 43/60 |
| 🥈 | Claude Opus 4.8 | 46.99 | 42/60 |
| 🥉 | Claude Sonnet 4.6 | 42.09 | 38/60 |
| 4 | GPT-5-mini | 35.23 | 32/60 |
| 5 | Claude Haiku 4.5 | 33.78 | 30/60 |
| 6 | Gemini 2.5 Pro | 31.06 | 28/60 |
| 7 | Llama 4 Maverick | 24.65 | 22/60 |
| — | human reference | 66.67 | — |
| Rank | Model | Objectives met |
|---|---|---|
| 🥇 | Claude Opus 4.8 | 5 / 8 |
| 🥈 | Claude Sonnet 4.6 | 3 / 8 |
| 🥉 | GPT-5 | 1 / 8 |
| — | Haiku 4.5 · GPT-5-mini · Gemini 2.5 Pro · Llama 4 Maverick | 0 / 8 |
Most of the gap to 66.67 is a correctness-gate effect, not a physical-quality shortfall: on tasks a model actually passes, its PPA sits within ~2.3% of the human reference on average. Four of seven models make no progress at all on agentic optimization. Cost does not track quality. See the paper for full per-tier results, the token-budget sensitivity analysis, and the reasoning behind reporting one condition over the other.
Want your model on the board? Open a model request —
the maintainer runs it through the official pinned flow and adds the result. Once public,
the live leaderboard is published at https://meetbhadra701-cloud.github.io/GateTruth/.
- Track A — RTL generation. The model receives a natural-language spec and a locked interface and emits one synthesizable SystemVerilog module, scored through the correctness gates and the PPA pipeline.
- Track B — agentic optimization. An agent iterates against the synthesis and timing tools — editing, re-linting, re-simulating, re-synthesizing — to meet a concrete objective (close timing, reduce area/power, remove latches) under an explicit token, time, tool-call, and spend budget, with a sequential-equivalence gate that forbids changing what the design computes.
A submission passes six stages. Stages 0–2 are correctness gates — each independently zeroes the score on failure (severability):
| Stage | Check | Tool |
|---|---|---|
| 0 | Lint | verilator --lint-only |
| 1 | Simulation vs. hidden vectors | Icarus + cocotb |
| 2 | Formal properties (where declared) | SymbiYosys |
| 3–5 | Synthesis, timing, power | Yosys + OpenSTA (sky130hd) |
Surviving designs are scored on the PPA geometric mean against the reference:
ppa = geomean(ref_area/area, ref_delay/delay, ref_power/power)
task_score = 100 * min(ppa, 1.5) / 1.5
The reference scores ppa = 1.0 → 66.67; the 100-point ceiling is a design 1.5×
better than the reference. Correctness is a gate, not a weight, so a model
cannot win by removing functionality.
Build the pinned image from the repository root:
docker build --platform linux/amd64 -t gatetruth:v1 -f flows/Dockerfile flowsScore a reference design through the Track A pipeline:
mkdir -p build/secure-src build/secure-output
git archive --format=tar HEAD | tar -xf - -C build/secure-src
chmod 0777 build/secure-output
docker run --rm --network none --cap-drop=ALL \
--security-opt no-new-privileges --memory=4g --pids-limit=512 --cpus=2 \
--mount "type=bind,src=$PWD/build/secure-src,dst=/work,readonly" \
--mount "type=bind,src=$PWD/build/secure-output,dst=/output" \
--workdir /work gatetruth:v1 ./gatetruth run \
--task t1_gray_counter \
--submission tasks/t1_gray_counter/ref/ref.sv \
--out /output/t1_gray_counter.jsonBuild the static leaderboard site from signed result manifests:
mkdir -p build/secure-output
chmod 0777 build/secure-output
docker run --rm --network none --cap-drop=ALL \
--security-opt no-new-privileges --memory=4g --pids-limit=512 --cpus=2 \
--mount "type=bind,src=$PWD/build/secure-src,dst=/work,readonly" \
--mount "type=bind,src=$PWD/build/secure-output,dst=/output" \
--workdir /work gatetruth:v1 ./gatetruth site \
--results results/eval-16384 --out /output/site-buildSee Secure execution for the canonical isolation
contract. The staged source tree contains no .git/, and execution containers
never receive provider API keys.
tasks/ 60 Track A task packages (spec, interface, reference, tests, constraints)
tasksB/ 8 Track B agentic task packages (baseline, objective, budget)
results/ signed per-sample manifests from every official campaign (Track A,
Track B, mutation certification, variance study) -- what the
Quickstart's `--results` flag and the paper's tables read from
harness/ evaluation harness — CLI, scoring, provider adapters, spend ledger
flows/ pinned Docker image + synthesis/timing/power flow scripts
scripts/ reproduction and contamination-scan utilities
site/ static leaderboard generator
paper/ full paper source (LaTeX + refs) and its auto-generated result tables
docs/ methodology notes
external-audit/ mutation-testing audit of external RTL benchmarks (read-only
vendor fetches, pinned commits, GateTruth-generated reports only)
Each task is a self-contained package: an original-prose spec (with a unique canary), a locked interface, a human-reviewed reference, public smoke + held-out hidden tests, formal properties where natural, and a timing constraint. Reference designs and hidden vectors require human sign-off; models never see them.
- Pinned flow. Every score is computed inside one image with fully version-locked tools and standard-cell library, so a result is a function of the design, not the tool versions.
- Canonical manifests. Each run emits a manifest with a SHA-256 integrity checksum over its canonical JSON; given the same pinned image, submission, and (for official runs) hidden vectors, re-running a design reproduces it byte-for-byte (tamper-evidence against accidental corruption, not a keyed authenticity signature) -- see "Reproducibility scope" below for exactly what that does and does not cover from a public checkout.
- Mutation-gated tests. New tasks are required to clear a ≥95% mutation-kill floor before admission, counting only behavioral simulation failures as kills. 14 of the 60 currently-admitted Track A tasks do not clear it today, a disclosed finding rather than a silently relaxed gate — see the paper for exactly which ones and why.
- Contamination controls. Per-task canaries, a sign-off-gated hidden-vector set, and a pre-publication canary scan.
- Reproducibility scope. A clean public checkout runs the public smoke tests, and
the reference reproduces its 66.67 by construction. Reproducing official model
scores additionally requires the maintainer-held hidden vectors (kept private for
contamination control), mounted via
GATETRUTH_HIDDEN_ROOT(the legacySILICONBENCH_HIDDEN_ROOTalias remains accepted).
New tasks, bug reports, and model requests are welcome — see CONTRIBUTING.md. Please read the Code of Conduct.
The paper (methodology, reference-suite design, and the full audit results) is
written and compiles cleanly from paper/ in this repository; an
arXiv submission is pending. Until it has a permanent identifier, please cite
the repository (see CITATION.cff).