ExBind

ExBind is a controlled diagnostic benchmark for visual-to-executable correspondence. It evaluates whether a model can map a visual or semantic referent to the exact structured reference required by an executable environment, and decomposes each prediction with deterministic annotations.

This repository is the public ExBind v2.0 release. It is a benchmark and evaluation package, not a correction, reranking, training, or agent-policy repository.

What is released

Component	Size	Purpose
Broad diagnostic suite	250 cases	Canvas, tree, DOM, graph, table, and SVG surface cases
Targeted structural suite	240 cases	120 table and 120 hierarchy cases
Paired latent groups	50 groups	SVG--canvas and tree--DOM consistency
Deterministic evaluators	--	Exact, dimension-wise, failure-stage, and cross-format scoring
Frozen paper outputs	--	Qwen2.5-VL-3B and Qwen3-VL-4B predictions and summaries
Supporting model outputs	--	LLaVA-Phi-3-mini broad/table predictions and summaries
Paper	--	ARR/arXiv manuscript source, PDF, and figures

The inherited EVG, SVGEditBench, and MiniWoB resources are not copied into this core release because their redistribution terms were not recoverable in the working checkout. Their converters, provenance notes, and bridge specification remain available in the source project and are documented here.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e .
export PYTHONPATH=src

Regenerate the controlled benchmark into a new directory:

python scripts/generate_broad.py --out outputs/reproduction/broad --seed 20260823
python scripts/generate_targeted.py --out outputs/reproduction/targeted --seed 20260824

Score a model prediction file without running inference:

python scripts/score_predictions.py \
  --records data/exbind_v2/broad/binding_records.jsonl \
  --predictions path/to/predictions.jsonl \
  --out outputs/scored/broad

The prediction contract is one JSON object per line. The required fields are case_id and prediction; the prediction payload contains only the surface reference, for example {"target_id": "canvas_obj_01"} or {"cell_id": "cell_r01_c02"}. The model is not required to emit reasoning or intermediate labels.

Public data layout

data/exbind_v2/
  broad/
  targeted/
  cross_format/
results/
  frozen_models/
  candidate_order/
  table_ablation/
huggingface/
  README.md
  data/
src/exbind/
scripts/
paper/
docs/

The huggingface/ directory is an upload-ready Hugging Face Dataset repository. It contains the same redistributable core records with a Dataset Card and explicit configurations. Upload it as a separate dataset repository; the GitHub repository remains the source-code and reproducibility home.

Reproducibility levels

1. Score release results: use the included prediction and score files; no model weights are required.
2. Regenerate benchmark records: run the deterministic generators with the fixed seeds in configs/exbind_v2.0-paper.yaml.
3. Rerun model inference: obtain the checkpoints separately and follow the model-specific terms and the strict output contract.
4. Rebuild the paper: compile paper/exbind_benchmark.tex with the included ACL style and figures.

See REPRODUCE_EXBIND.md, the frozen inference manifest, and the release manifest for exact paths and hashes.

Scope and licenses

Core software is covered by the scoped MIT license in LICENSE. Newly generated core records are covered by the scoped CC BY 4.0 terms in DATA_LICENSE. Model weights and inherited benchmark data are not included; see THIRD_PARTY_NOTICES.md before obtaining or redistributing them.

Citation

Please cite the ExBind paper included under paper/ and identify the release as ExBind v2.0 in reported results.