StructureBench

StructureBench is a comprehensive plug-and-play benchmark suite for evaluating structured generation on resource-constrained edge devices. It brings together over 11 publicly-available datasets that span six representative task families—information extraction, tool calling, SQL/code completion, mathematical reasoning, scientific report generation and DSL synthesis—covering both text-only and vision-language inputs. For every task we release unified prompt templates, machine-checkable grammars and automatic evaluators, so that results obtained with different models or decoders remain directly comparable.

Click here jump to our leaderboard

Read the supplementary materials / appendix

The suite currently supports three state-of-the-art constrained decoding frameworks—Outlines, XGrammar and Guidance—alongside a prompt-only baseline, and ships with wrappers that make switching between them as simple as passing a --mode flag. On the model side, StructureBench targets compact language and vision-language models between 0.5 B and 8 B parameters, a regime where instruction-following is brittle and structural guarantees matter most.

StructureBench logs metrics at three complementary levels: (i) syntactic compliance, (ii) task-level semantic accuracy and (iii) the marginal gain in semantics attributable exclusively to constraints within the space of valid outputs. Additional telemetry captures runtime latency, memory footprint and token-level trace files to support fine-grained diagnostics.

All components are lightweight and self-contained—the entire benchmark, including datasets, grammars and evaluation scripts, runs on a single consumer-grade GPU laptop and can be extended with a single YAML entry. We publish baseline results for 12 open-source models and provide best-practice recipes that map application scenarios to reliable model–decoder combinations, empowering developers to ship trustworthy on-device structured generation out of the box.

Environment & Dependencies

• Python 3.10 recommended; typical runs use a CUDA GPU (install a matching GPU wheel for torch/transformers).
• Core runtime: torch (GPU build), transformers, accelerate, datasets, tqdm, filelock, sentencepiece, pillow (for VLM).
• XGrammar constrained decoding (optional): xgrammar (install from PyPI or pip install -e xgrammar to use the bundled source).
• BFCL extras (imports needed even in prompt-mode): tree_sitter, tree_sitter_java, tree_sitter_javascript, anthropic, cohere, openai, google-genai, mistralai, writerai, qwen-agent, datamodel-code-generator, boto3.
• You can install all dependencies with:

  pip install -r requirements.txt

• Install dependencies you need (manually pick the version that matches your machine/CUDA toolkit):

  # torch GPU wheel (pick the right cuXXX index for your driver)
  pip install torch==2.7.1+cu118 --index-url https://download.pytorch.org/whl/cu118
  pip install transformers==4.48.3 tokenizers==0.21.0 huggingface-hub==0.26.2 accelerate==1.12.0 datasets sentencepiece pillow filelock tqdm
  pip install tree_sitter==0.21.3 tree_sitter_java==0.21.0 tree_sitter_javascript==0.21.0
  pip install anthropic cohere openai google-genai mistralai writerai qwen-agent datamodel-code-generator boto3

Generation (text)

Module: data/text_models_generate.py

CLI arguments:

• --model_name_or_path (str, required)
• --device (cuda|cpu, default cuda)
• --no_trust_remote_code (flag)
• --torch_dtype (default float16)
• --attn_implementation (str, optional)
• --max_new_tokens (int, default 256)
• --temperature (float, default 0.2)
• --top_p (float, default 1.0)
• --repetition_penalty (float, default 1.1)
• --frequency_penalty (float, default 0.2)
• --dataset (str, required; registered in data/prompts.py)
• --data_file (optional override)
• --split (default test)
• --num_samples (int, per-dataset semantics; BFCL interprets as per-category cap)
• --output_file (str, required)
• --prompt_kwargs (JSON string forwarded to prompt loader)
• Defaults: loader full-scope behavior (e.g., BFCL) when not set.
• --mode (prompt|xgrammar|guidance|outlines|llama_cpp, default prompt)
• --constraint_template (path to .json/.cfg/.ebnf/.regex constraints; used by xgrammar and other constrained backends)
• --think (off|on, Qwen3 think handling)
• --append_output (flag; default overwrite)
• --store_prompt (flag; default off)
• --verbose / --quiet (default verbose on; use --quiet to disable)

Examples:

# Generic dataset
python -m data.text_models_generate \
  --dataset humaneval \
  --model_name_or_path <your_model> \
  --output_file outputs/humaneval_pred.jsonl \
  --num_samples 10 --store_prompt

# BFCL single-turn prompt-mode (full scope, 2 per category)
python -m data.text_models_generate \
  --dataset bfcl \
  --model_name_or_path facebook/opt-125m \
  --output_file outputs/bfcl_pred.jsonl \
  --num_samples 2 \
  --prompt_kwargs "{\"test_categories\":[\"simple_python\",\"simple_java\",\"simple_javascript\",\"multiple\",\"parallel\",\"parallel_multiple\",\"live_simple\",\"live_multiple\",\"live_parallel\",\"live_parallel_multiple\"]}" \
  --store_prompt --device cpu

# FOLIO (CoT + 1-shot prompt)
python -m data.text_models_generate \
  --dataset folio \
  --model_name_or_path <your_model> \
  --output_file outputs/folio_pred.jsonl \
  --split validation \
  --num_samples 5 \
  --store_prompt

Constrained generation (xgrammar)

Dependencies:

• Install xgrammar (PyPI) or the bundled source (pip install -e xgrammar).

Usage:

• Add --mode xgrammar and --constraint_template <path>.
• Supported templates: .json (JSON Schema), .ebnf (EBNF grammar), .regex.
• Templates live under constraint_templates/.

Example:

python -m data.text_models_generate \
  --dataset text2sql \
  --model_name_or_path <your_model> \
  --output_file outputs/text2sql_pred.jsonl \
  --mode xgrammar \
  --constraint_template constraint_templates/sql/sql.ebnf

Trace generation

Trace scripts under trace_generator/ run greedy stepwise decoding with xgrammar masks and log per-token decisions to JSONL.

Example:

python -m trace_generator.humaneval_trace_generate \
  --model_name_or_path <your_model> \
  --num_samples 2 \
  --trace_file trace_generator/logs/humaneval_trace.jsonl

Available scripts:

• trace_generator.humaneval_trace_generate
• trace_generator.amr_trace_generate
• trace_generator.planetarium_trace_generate
• trace_generator.latexocr_trace_generate
• trace_generator.smileseval_trace_generate
• trace_generator.text2sql_trace_generate
• trace_generator.bfcl_trace_generate

Common flags:

• --data_file, --split, --num_samples
• --max_new_tokens, --temperature, --top_p
• --repetition_penalty, --frequency_penalty
• --constraint_template, --trace_file, --repeat_window
• plus model flags (--device, --torch_dtype, --attn_implementation, --no_trust_remote_code)

Evaluation

Use dataset-specific evaluators under eval/. Generic fallback: eval/x_eval.py.

BFCL single-turn (prompt-mode)

Module: eval/bfcl_eval

CLI arguments:

• --pred_file (required)
• --summary_file (optional; defaults to eval/bfcl_eval_summary.json)
• --data_root (default D:\StructureBench\datasets\bfcl_eval\data)
• --test_categories (optional list/group; always limited to allowed scope)
• --num_samples (per-category cap; 0 = all)
• --underscore_to_dot (flag to normalize function names)
• --verbose / --quiet (default verbose on)

Example:

python -m eval.bfcl_eval \
  --pred_file outputs/bfcl_pred.jsonl \
  --summary_file eval/bfcl_eval_summary.json \
  --num_samples 2

FOLIO (Prover9)

Module: eval/folio_eval.py

Example:

python -m eval.folio_eval \
  --pred_file outputs/folio_pred.jsonl \
  --output_file eval/folio_eval.jsonl \
  --summary_file eval/folio_eval_summary.json \
  --split validation

Defaults: data_file -> datasets/FOLIO/folio_v2_validation.jsonl (choose train/validation by --split). The evaluator auto-sets PROVER9/MACE4 to the bundled binaries under eval/prover9/bin.

Generic alignment check

python -m eval.x_eval \
  --dataset <name> \
  --pred_file <pred.jsonl> \
  --output_file eval/<name>_eval.jsonl \
  --summary_file eval/<name>_summary.json \
  --num_samples 10

Other dataset-specific evaluators

See eval/*_eval.py for per-dataset flags; most accept --pred_file, --output_file, --data_file, --split, --num_samples, --verbose.

Files of Interest

• common/: shared CLI/model helpers (cli.py, text_model_utils.py).
• data/: centralized prompt loaders (prompts.py, bfcl_prompts.py), generic generators (text_models_generate.py, vl_models_generate.py).
• eval/: per-dataset evaluators and logging (bfcl_eval/, x_eval.py, eval_logging.py).

Dataset layout and overrides

By default, datasets live under datasets/ and are referenced by data/prompts.py. Use --data_file in generators/evaluators to point to a local file.

BFCL specifics:

• Prompt loader: pass --prompt_kwargs '{"data_root":"/path/to/bfcl_eval/data"}'.
• Evaluator: pass --data_root /path/to/bfcl_eval/data.

Dataset keys and default folders:

Dataset key	HF dataset repo	Default local folder
amr	hoshuhan/amr-3-parsed	datasets/amr-3-parsed
ape21	MU-NLPC/Calc-ape210k	datasets/Calc-ape210k
bigmath	SynthLabsAI/Big-Math-RL-Verified	datasets/Big-Math-RL-Verified
bfcl	(bundled)	eval/bfcl_eval/data
chequessample	shivalikasingh/cheques_sample_data	datasets/cheques_sample_data
folio	yale-nlp/FOLIO	datasets/FOLIO
hermes	NousResearch/hermes-function-calling-v1	datasets/hermes-function-calling-v1
humaneval	openai/openai_humaneval	datasets/openai_humaneval
jsschemabench	epfl-dlab/JSONSchemaBench	datasets/JSONSchemaBench
latexocr	linxy/LaTeX_OCR	datasets/LaTeX_OCR
planetarium	BatsResearch/planetarium	datasets/planetarium
smileseval	eth-sri/smiles-eval	datasets/smiles-eval
text2sql	gretelai/synthetic_text_to_sql	datasets/synthetic_text_to_sql