Reproducible multimodal evaluation for semantic counting in document images

arXiv | Project | Dataset | ADOPD2026 Code | ADOPD2026 Dataset

DocCount evaluates whether a vision-language model can count entities that satisfy an explicit semantic definition in a document image. The target is not generic object counting: each question defines what belongs to the requested class, what must be excluded, and how multiple visual components should be grouped into one instance.

This repository provides one evaluation path for both self-hosted vLLM models and remote OpenAI-compatible chat-completions endpoints. It sends the released prompt verbatim with the original image bytes, preserves the complete visible model text and response metadata, and computes exact integer accuracy over the full benchmark denominator.

At A Glance

442 questions	442 original images	4 semantic classes	2,555 counted instances
one question per page	no image resizing	exact integer answers	full-denominator accuracy

Repository Layout

Path	Purpose
src/doccount_eval/	Dataset loading, multimodal requests, retries, resumption, answer parsing, merging, and scoring
prompts/counting_protocol.md	Exact benchmark prompt template and final-answer contract
scripts/serve_vllm.sh	Generic vLLM server launcher for one-image chat requests
scripts/evaluate_local_vllm.sh	Evaluation against a local vLLM endpoint
scripts/evaluate_remote_api.sh	Evaluation against a remote OpenAI-compatible endpoint
tests/	Protocol, payload, request handling, and scoring tests

The repository contains no model weights. Self-hosted evaluation works with a checkpoint supported by the installed vLLM version; remote evaluation uses a model identifier exposed by the selected endpoint.

Evaluation Protocol

Each row of adopd/DocCount contains an original JPEG, an explicit class definition, a counting question, the complete model prompt, and an integer answer. The evaluator sends exactly:

1. the embedded original image bytes as one multimodal image_url data part;
2. the row's prompt string as one text part.

It does not resize or recompress the image, add a system message, rewrite the class definition, or append hidden prompt text. Before a request, the image bytes are checked against the released SHA-256 value.

The common prompt form is:

Target class: {target_class}

Definition:
{class_definition}

Question:
{question}

Reason about the image carefully before answering. You should include a detailed rationale before the final answer.

The final line must contain exactly one integer in this format:
<answer>N</answer>

The primary parser requires exactly one answer tag and requires it to be the final non-whitespace content. Request errors, missing responses, and parse failures count as incorrect. The primary score is:

exact-count accuracy = exactly correct answers / 442

An optional last-integer fallback exists for diagnostic studies, but results using it must not be reported as the primary protocol.

Installation

Python 3.10 or newer is required. With uv:

uv venv --python 3.10 .venv
source .venv/bin/activate
uv pip install -e .

Set the Hugging Face cache location when the default location is unsuitable:

export HF_HOME="$HOME/.cache/huggingface"

Optionally download the dataset in native repository layout:

hf download adopd/DocCount --repo-type dataset --local-dir data

Validate the complete local release without calling a model:

doccount-evaluate \
  --backend api \
  --model org/model-name \
  --data-files 'data/test-*.parquet' \
  --dry-run

Omit --data-files to load adopd/DocCount directly from the Hub.

Remote API

Export credentials rather than placing them in commands, logs, or checked-in environment files:

export MODEL_API_BASE_URL="https://api.example.com/v1"
read -rsp "API key: " MODEL_API_KEY; export MODEL_API_KEY; echo
export MODEL="provider/model-name"
export RUN_NAME="model-name__reasoning-on"

bash scripts/evaluate_remote_api.sh
doccount-score --run-root "runs/${RUN_NAME}"

Defaults are temperature=1, top_p=0.95, max_tokens=32768, 16 concurrent requests, a 600-second request timeout, and four retries after the initial attempt. Override them through environment variables:

CONCURRENCY=32 MAX_TOKENS=32768 \
  bash scripts/evaluate_remote_api.sh

Optional request fields

Some endpoints expose additional request fields. The evaluator does not infer these from a model name; documented fields can be supplied with --extra-body-json. They are written to run_config.json, while secrets stay in headers or environment variables.

Self-Hosted vLLM

Install a vLLM build that supports the selected model and accelerator. Start the server in one terminal:

export MODEL="org/model-name"
bash scripts/serve_vllm.sh

The launcher intentionally does not prescribe model placement, parallelism, context length, or memory settings. When a checkpoint needs additional vLLM arguments, append the options documented for that checkpoint to the launcher command.

The evaluator sends each image as an in-memory data URL and does not require the server to read benchmark files from the local filesystem.

Run the evaluator from a second terminal:

export MODEL="org/model-name"
export RUN_NAME="model-name__self-hosted"

CONCURRENCY=8 bash scripts/evaluate_local_vllm.sh

doccount-score --run-root "runs/${RUN_NAME}"

CONCURRENCY controls simultaneous evaluation requests independently of how the model server is deployed.

Resumption And Failures

The evaluator appends one JSON object per completed row and flushes it immediately. Re-running the same command skips rows whose latest record has both status="ok" and a parsed answer. Request errors, data errors, and unparseable responses remain eligible for retry. --overwrite deliberately re-evaluates every selected row.

Transient network errors, timeouts, HTTP 408/409/425/429 responses, server errors, malformed response JSON, and empty response content use bounded exponential backoff with jitter. Non-retryable client errors are recorded without repeatedly sending the same request. Add --retry-parse-failures only when repeated sampling after a formatting failure is part of the declared evaluation setup.

Outputs

runs/<run-name>/
  run_config.json
  responses.partial.jsonl
  responses.jsonl
  scores/
    summary.json
    summary.md

Each response row keeps the benchmark identifiers and ground truth together with:

• model, backend, sampling settings, and timestamps;
• full visible response_text and separately exposed reasoning_text;
• parsed answer, parse status, exact correctness, and finish reason;
• latency, token usage, response ID, attempt count, and retry errors;
• raw text and metadata from earlier response attempts when parse retries are enabled;
• explicit request_error or data_error records when generation fails.

doccount-score selects the latest record for each sample_id, writes one merged responses.jsonl row for every benchmark question, and reports overall and per-class accuracy. Missing rows stay visible and remain in the denominator.

Results

The chart below summarizes one complete 442-question run for each model. All entries use exact-count accuracy over the full benchmark denominator. Model configurations differ, so the values are reference results rather than a leaderboard.

Model	Correct	Exact accuracy
Kimi K2.5	322 / 442	72.85%
Qwen3.6 35B A3B	319 / 442	72.17%
Qwen3.5 397B A17B	313 / 442	70.81%
GPT-5.5	304 / 442	68.78%
Claude Sonnet 4.5	301 / 442	68.10%
GPT-5.2	293 / 442	66.29%
GLM-4.6V	285 / 442	64.48%
Gemma 4 31B IT	269 / 442	60.86%

Development

Install test dependencies and run the release tests:

uv pip install -e '.[dev]'
pytest -q

Tests cover the strict answer contract, explicit fallback behavior, multimodal payload ordering, endpoint normalization, reserved request fields, request retry behavior, and full-denominator scoring.

License

The source code in this repository is licensed under the Apache License 2.0. The DocCount dataset is distributed separately under CC BY-NC-ND 4.0. See the dataset card for intended use and limitations. Model checkpoints remain subject to their own licenses.

Citation

@misc{zhu2026thinkinganchorsgroundedefficient,
  title={Thinking with Anchors: Grounded and Efficient Document Reasoning}, 
  author={Sichen Zhu and Yuchen Zhu and Wenzhuo Xu and Jason Kuen and Wanrong Zhu and Jing Shi and Xuan Shen and Quanyi Wang and Yiwei Wang and Yujun Cai and Bing Shuai and Qin Zhang and Yongxin Chen and Shilong Liu and Molei Tao and Jiuxiang Gu},
  year={2026},
  eprint={2608.04424},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.04424}, 
}