EAHC: Execution-Anchored Hallucination Calibration

Separate the Wheat from the Chaff: Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation

---

🎯 Overview

Large Language Models (LLMs) have shown remarkable capabilities in code generation, but their performance on low-resource Hardware Description Languages like Verilog lags significantly behind general-purpose languages. While sampling multiple candidates (Pass@k) substantially improves the likelihood of generating correct solutions, automatically selecting the optimal candidate from k alternatives remains an open challenge.

EAHC (Execution-Anchored Hallucination Calibration) addresses this code reranking problem through a dual-channel framework that:

1. EAHC-R (Reasoning Channel): A 4B reasoning discriminator fine-tuned on 47K compiler-verified judgment traces via multi-teacher distillation
2. EAHC-T (Execution Channel): A testbench generator leveraging RAG over a 53K corpus for execution verification
3. Hierarchical Selection: A fusion mechanism that anchors reasoning signals to execution evidence, eliminating reasoning hallucination

Key Insights

Through systematic empirical study across 9 code generation models and 2 benchmarks, we identify two critical limitations in existing approaches:

• Poor Domain Transferability: Existing reranking methods designed for general-purpose languages fail to transfer to Verilog's hardware-specific constructs
• Reasoning Hallucination in LLM-as-a-Judge: LLMs produce inconsistent correctness judgments for functionally equivalent code

Our theoretical analysis proves that:

1. Joint use of execution and reasoning signals outperforms either alone (Theorem 1)
2. Acquiring signals independently strictly outperforms interactive fusion where the reasoner observes execution feedback (Theorem 2)

---

✨ Key Features

• State-of-the-Art Performance:

  • On VerilogEval-v2: Elevates average Pass@1 from 53.99% to 65.10%, recovering 64.2% of the gap to Pass@10 oracle (71.30%)
  • On ResBench: Improves Pass@1 from 53.18% to 68.25%, recovering 61.3% of the gap to oracle (77.78%)

• Orthogonal to Training Methods:

  • Provides consistent gains of 8.93% to 19.64% across base, SFT, and RL-optimized models
  • Model-agnostic enhancement that works with any code generation model

• Comprehensive Baseline Comparisons:

  • Probability-based: Generation probability, CodeReviewer
  • Semantic matching: CodeRank (with Qwen3-Embedding and Jina-Code-v2)
  • Execution-based: CodeT (Self-generated and GPT-generated testbenches)
  • LLM-as-a-Judge: Code-DiTing (1.5B and 7B variants)

---

📂 Repository Structure

verilog_rerank_yosys/
├── verilogeval_v2/              # VerilogEval-v2 benchmark experiments
│   ├── problems_verilogeval_v2.jsonl      # Problem definitions
│   ├── results/                            # Model generation results
│   │   ├── pass1_*.json                   # Greedy decoding results
│   │   └── pass10_*.json                  # Sampling results (k=10)
│   ├── rerank_results/                    # Reranking results
│   │   ├── EAHC_R_reasoning_*.json       # EAHC-R results
│   │   ├── EAHC_T_testbench_*.json       # EAHC-T results
│   │   └── hybrid_*_a*.json              # Full EAHC results with different α
│   ├── functional_correctness.py          # Evaluation script
│   ├── generate_api.py                    # Code generation via API models
│   ├── generate_llm.py                    # Code generation via local models
│   └── generate_codev_r1.py              # CodeV-R1 specific generation
│
├── resbench/                    # ResBench benchmark experiments
│   ├── problems_resbench.jsonl           # Problem definitions
│   ├── results/                          # Model generation results
│   ├── rerank_results/                   # Reranking results (same structure as VerilogEval-v2)
│   ├── functional_correctness.py         # Evaluation script
│   ├── generate_api.py                   # Code generation via API models
│   ├── generate_llm.py                   # Code generation via local models
│   └── generate_codev_r1.py             # CodeV-R1 specific generation
│
├── results_reasoning/           # Reasoning judgments from EAHC-R
│   └── EAHC_R_*.pkl             # Pickled reasoning results
│
├── results_testbench/           # Generated testbenches from EAHC-T
│   └── EAHC_T_*.pkl             # Pickled testbench results
│
├── ours_using_reasoning.py      # EAHC-R reranking implementation
├── ours_using_testbench.py      # EAHC-T reranking implementation
├── reasoning_with_testbench.py  # Full EAHC (fusion) implementation
│
├── save_reasoning.py            # Save EAHC-R reasoning judgments
├── save_testbench.py            # Save EAHC-T testbenches
├── run_save_reasoning.sh        # Batch script for reasoning generation
│
├── baseline_probability.py      # Probability-based baseline
├── baseline_codereviewer.py     # CodeReviewer baseline
├── baseline_coderank.py         # CodeRank baseline
├── baseline_codet.py            # CodeT baseline
├── baseline_code_diting.py      # Code-DiTing baseline
├── baseline_vcd_rnk.py          # VCD-based ranking baseline
│
├── codet_gen_testcases_api.py   # Generate testcases for CodeT (API models)
├── codet_gen_testcases_llm.py   # Generate testcases for CodeT (local models)
│
├── compute_mtest.py             # Compute McNemar's test statistics
│
└── readme.md                    # This file

---

---

🚀 Usage

Note on Trained Models and Datasets:

• 🔒 Trained models (EAHC-R discriminator, EAHC-T generator) and training datasets (VeriJudge-47K, VeriTest-53K) will be open-sourced after paper acceptance.
• ✅ Currently available: Complete project code and experimental result data (see results/, rerank_results/ folders).

1. Code Generation

Generate k=10 candidates for each problem:

# Using API-based models (GPT-5, DeepSeek-V3, GLM-4.6)
python verilogeval_v2/generate_api.py --model_name gpt-5 --k 10

# Using local models (Qwen2.5-Coder, OpenCoder, Seed-Coder, etc.)
python verilogeval_v2/generate_llm.py --model_name Qwen2.5-Coder --k 10

2. Baseline Reranking

Run baseline reranking methods:

# Probability-based
python baseline_probability.py --dataset verilogeval_v2 --model_name Qwen2.5-Coder

# CodeReviewer
python baseline_codereviewer.py --dataset verilogeval_v2 --model_name Qwen2.5-Coder

# CodeRank
python baseline_coderank.py --dataset verilogeval_v2 --model_name Qwen2.5-Coder --embedding_model qwen3

# CodeT
python codet_gen_testcases_api.py --dataset verilogeval_v2 --testbench_model gpt-5
python baseline_codet.py --dataset verilogeval_v2 --model_name Qwen2.5-Coder

# Code-DiTing
python baseline_code_diting.py --dataset verilogeval_v2 --model_name Qwen2.5-Coder --judge_model DiTing-7B

3. EAHC Reranking

Step 1: Generate reasoning judgments (EAHC-R)

# Save reasoning judgments
python save_reasoning.py --dataset verilogeval_v2 --model_name CodeV-R1-RL --n 3

# Run EAHC-R reranking
python ours_using_reasoning.py --dataset verilogeval_v2 --model_name CodeV-R1-RL --n 3

Step 2: Generate testbenches (EAHC-T)

# Save testbenches
python save_testbench.py --dataset verilogeval_v2 --model_name CodeV-R1-RL --k 5

# Run EAHC-T reranking
python ours_using_testbench.py --dataset verilogeval_v2 --model_name CodeV-R1-RL --k 5

Step 3: Full EAHC (Fusion)

# Run hierarchical fusion with α=0.6 (default)
python reasoning_with_testbench.py \
    --dataset verilogeval_v2 \
    --model_name CodeV-R1-RL \
    --n 3 \
    --k 5 \
    --alpha 0.6

Batch Processing: Use the provided script to run all models:

bash run_save_reasoning.sh

4. Evaluation

Compute Pass@1 accuracy:

python verilogeval_v2/functional_correctness.py \
    --input_file verilogeval_v2/rerank_results/hybrid_CodeV-R1-RL_3_a0.6.json

---

📊 Datasets

VerilogEval-v2

• Size: 156 problems
• Source: VerilogEval
• Features: Human-written specifications, ground-truth testbenches
• Task Types: RTL design, FSM, arithmetic modules, etc.

ResBench

• Size: 56 problems
• Source: ResBench
• Features: More complex, realistic hardware designs
• Task Types: Resource-intensive designs (multipliers, dividers, etc.)

Dataset Format (.jsonl):

{
  "task_id": "example_001",
  "prompt": "Design a 4-bit counter with synchronous reset...",
  "module_header": "module counter(input clk, input rst, output reg [3:0] count);",
  "test": "// Testbench code...",
  "canonical_solution": "// Reference solution..."
}

---

📈 Experimental Results

Main Results (Average Pass@1)

Method	VerilogEval-v2	ResBench
Original Pass@1	53.99%	53.18%
CodeT-GPT (Best Baseline)	59.19%	63.29%
EAHC (Ours)	65.10%	68.25%
Pass@10 (Oracle)	71.30%	77.78%
Gap Recovery	64.2%	61.3%

Model-Specific Results on VerilogEval-v2

Model	Pass@1	EAHC	Improvement
GPT-5	85.90%	83.97%	-
DeepSeek-V3	73.08%	81.41%	+8.33%
GLM-4.6	76.28%	81.41%	+5.13%
Qwen2.5-Coder	33.33%	46.15%	+12.82%
OpenCoder	31.41%	50.00%	+18.59%
Seed-Coder	47.44%	62.18%	+14.74%
HaVen	40.38%	53.85%	+13.47%
VeriPrefer	41.67%	60.26%	+18.59%
CodeV-R1	56.41%	66.67%	+10.26%

Ablation Study

Component	VerilogEval-v2	ResBench
EAHC-T only (k=3)	57.98%	62.50%
EAHC-R only (n=3)	62.77%	63.10%
EAHC (Full)	65.10%	68.25%

Orthogonality with Training Methods (CodeV Family)

Training Stage	Pass@1	EAHC	Improvement
Base (Qwen2.5-Coder)	33.33%	46.15%	+12.82%
SFT (CodeV-R1)	56.41%	66.67%	+10.26%
RL (CodeV-R1-RL)	69.23%	78.21%	+8.98%

Independent vs. Interactive Fusion

Fusion Strategy	VerilogEval-v2	ResBench
Interactive Fusion	60.61%	63.69%
Independent Fusion (Ours)	65.10%	68.25%
Improvement	+4.49%	+4.56%

---

🔬 Key Contributions

1. Problem Formulation and Empirical Findings:

   • First systematic empirical study on Verilog code reranking across 9 models and 2 benchmarks
   • Identified two critical limitations: poor domain transferability and reasoning hallucination

2. Framework with Theoretical Grounding:

   • Dual-channel architecture with independent signal acquisition
   • Theoretical proof that independent fusion outperforms interactive fusion
   • Compiler-in-the-loop distillation for high-quality training data

3. State-of-the-Art Performance:

   • Best reranking performance across all 18 configurations (9 models × 2 benchmarks)
   • Recovers over 60% of the Pass@1 to Pass@10 gap
   • Orthogonal to training-based optimizations

---

🙏 Acknowledgments

We thank the authors of VerilogEval and ResBench for providing the benchmarks, and the developers of the baseline methods for making their code available.

---

🔔 Data Availability

• ✅ Available Now: Complete project code and experimental result data

  • All scripts for baselines and EAHC
  • Generated results on VerilogEval-v2 and ResBench
  • Reranking results for all configurations

• 🔒 Coming After Paper Acceptance:

  • EAHC-model: Fine-tuned Qwen3-4B model
  • VeriJudge-47K: Reasoning judgment training dataset
  • VeriTest-53K: Testbench generation training dataset

Stay tuned for the full release upon paper acceptance!

---