PR-Bench is a referring expression comprehension and grounding benchmark designed to evaluate whether multimodal large language models can bridge fine-grained visual perception and compositional semantic reasoning. It contains 6,000 carefully curated image-expression pairs across six subcategories, with 1,000 samples per category.
-
[2026-07-25] We released the official PR-Bench evaluation code and evaluation data, including evaluation scripts, images, and annotations. We welcome researchers to evaluate their models on PR-Bench.
-
[2026-07-23] We launched the official PR-Bench leaderboard for tracking and comparing model performance on the benchmark.
- [2026-07-11] Our paper "Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding" has been accepted by ACM Multimedia 2026 (ACM MM 2026).
Multimodal Large Language Models have made great progress in grounding tasks, yet existing methods still struggle to unify precise localization and complex reasoning. Text-based grounding methods often rely on coordinate or index prediction, which limits dense object perception, while latent token-based grounding methods may lack explicit spatial references and interpretable reasoning steps.
PR-Bench is designed to evaluate this perception-reasoning gap in referring expression comprehension. It focuses on challenging grounding cases where models must identify the correct visual target from a natural-language expression or reject an invalid reference when the described target is absent.
The benchmark is organized around two core dimensions:
- Visual-Cue Perception, which evaluates direct visual grounding from observable cues.
- Compositional Reasoning, which evaluates grounding from relational, contextual, or invalid references.
These two dimensions are further divided into six mutually exclusive subcategories. The three perception-oriented tasks mainly test whether models can use explicit visual evidence, while the three reasoning-oriented tasks require stronger relational, contextual, or rejection-aware understanding.
- Attribute: focuses on intrinsic and directly observable visual properties of objects, such as color, texture, material, shape, and state. Target identification relies on distinguishing fine-grained visual cues from same-category distractors.
- Position: captures spatial relationships between different objects in an image. It requires localizing a target based on its relative placement to heterogeneous anchor objects in the scene.
- Interaction: focuses on the spatial arrangement and relative ordering among objects of the same category. It typically relies on ordinal or relative positioning within a homogeneous group.
- Relation: identifies targets via comparative relationships with distinct anchor objects, leveraging shared or contrasting visual attributes. This category requires grounding multiple entities and then performing comparative judgment.
- Commonsense: identifies targets through contextual or functional descriptions instead of explicit visual attributes. The referring expression describes the target by its use, purpose, or context in the scene.
- Rejection: focuses on handling negation and non-existent references. The expression contains one or more conditions that do not match any object in the image, requiring the model to reject invalid references without hallucinating a target.
PR-Bench uses high-resolution images and emphasizes visually rich scenes with diverse objects. The benchmark is built from images sourced from FineHARD and refined through an MLLM-assisted annotation pipeline followed by multi-round human verification.
PR-Bench contains 6,000 referring expression grounding samples over 3,102 images. Each category contains 1,000 samples.
| Type | Images | Phrases | Avg. Image Size | Avg. Box Size | Avg. Phrase Len. |
|---|---|---|---|---|---|
| Attribute | 701 | 1,000 | 1602×1301 | 10.76% | 9.3 |
| Position | 735 | 1,000 | 1588×1294 | 8.34% | 11.0 |
| Interaction | 642 | 1,000 | 1608×1292 | 9.43% | 10.3 |
| Relation | 695 | 1,000 | 1619×1282 | 6.15% | 14.4 |
| Commonsense | 715 | 1,000 | 1627×1280 | 6.14% | 15.5 |
| Rejection | 764 | 1,000 | 1629×1301 | N/A | 13.8 |
| Total | 3,102 | 6,000 | 1612×1292 | 8.16% | 12.4 |
All PR-Bench data are available on the PR-Bench Hugging Face dataset page. Please visit the page to download the complete dataset.
PR-Bench evaluates model predictions for referring expression grounding and rejection. The detailed prediction format and scripts are provided in evaluation/README.md.
A typical workflow is:
- Run your model on
data/annotation.jsonl. - Save model predictions in JSONL format.
- Evaluate the prediction file using the provided evaluation script.
cd PR-Bench/evaluation
python get_prediction_Acc.py --prediction_file path/to/your_prediction_file.jsonlThe repository also includes Qwen3-VL example inference scripts in evaluation/. For other models, adapt the model loading, inference, and response parsing code.
Visit the PR-Bench Leaderboard to view the latest results.
We warmly invite the community to evaluate their models on PR-Bench. If you want to add your model to the leaderboard or share results with us, please send the model responses and evaluation results by email.
Please include:
- model name and version;
- model size and checkpoint source if applicable;
- prediction file in
.jsonlformat; - evaluation log or final scores;
- brief description of the inference setting;
- paper, technical report, or project link if available.
Please send submissions and questions to:
tyigao@stu.xjtu.edu.cn
Submitted results may be included in the public leaderboard or project page to support transparent and reproducible benchmarking.
For questions about PR-Bench data, evaluation, or result submission, please contact:
tyigao@stu.xjtu.edu.cn
The benchmark annotations and evaluation resources are licensed under the Creative Commons Attribution-NonCommercial 4.0 International License.
The benchmark is intended for non-commercial research and educational purposes. If you use this benchmark, its annotations, or evaluation tools in your work, please cite our paper:
@article{gao2026motto,
title = {Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding},
author = {Gao, Tianyi and Fang, Han and Ding, Tianyi and Li, Hao and Wei, Xin and Sun, Hongbo and Dong, Xiaodong and Yuan, Ye and Xu, Jinglin and Liang, Kongming and Sun, Hao and Xin, Jingmin},
journal = {arXiv preprint arXiv:2607.24407},
year = {2026}
}The original images remain subject to the licenses and terms of their respective source datasets.