OpenVLHarness

Turn any MLLM into an evidence-based visual reasoner that thinks and codes in visual space.

• Training-free. Works with the backbone's existing weights through native tool calling; nothing is fine-tuned.
• Nine capabilities. Visual grounding, OCR, depth, camera trajectory, zoom, text search, image search, webpage visits and a Python coding agent, each behind a task-level interface that returns evidence the model can read.
• Multimodal memory. Images, boxes, masks, depth maps and web references stay addressable for the whole question, so later tools and code can build on them.
• Consistent gains. Across 22 benchmarks in four domains, seven backbones gain 9.7 to 12.6 points on average over the same models without tools.
• Any registered model. Presets for seven of the paper's eight backbones, plus every Qwen3-VL Instruct size and OpenAI's GPT-5 to GPT-6.1 models in the model registry.

• How it works
• Results
• Installation
• Quick start
• Evaluation
• Configure and extend
• Repository layout
• Documentation and tests
• Acknowledgements
• License
• Citation

The model chooses tools, inspects their evidence, and continues until it can answer. Three components let it combine evidence across steps:

• Capability layer: nine task-level interfaces turn tool outputs into readable observations, including annotated images, summaries, and compact numerical data.
• Multimodal memory: images, structured data, and web references remain addressable throughout a question. Each new question starts with fresh memory.
• Python Coding Agent: the backbone writes code in a separate context to calculate over stored boxes, masks, depth maps, and other tool results.

For example, an agent can locate objects, estimate depth, and compute their spatial relationship; or crop an unfamiliar object, search for visual matches, and verify the answer on a webpage.

Every preset uses Qwen3-VL-8B-Instruct for multimodal search summarization and webpage extraction, independently of the main model. The coding agent uses the same backbone as the main model.

Average score over the paper's 22 benchmarks (counting and grounding, multimodal search, general VQA and hallucination, and spatial understanding):

Qwen3-VL results are the mean of three runs. Per-benchmark results, comparisons with other harnesses, and recorded trajectories are on the project page.

Use Python 3.10 or newer. From a source checkout:

git clone https://github.com/uclanlp/OpenVLHarness.git
cd OpenVLHarness
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[qwen,server]'

This installs the agent, Qwen tokenizer support, and the tool router. The model servers have separate environments and dependencies. For a client connecting to an existing router, pip install -e . is sufficient; add the qwen extra when using Qwen for local context counting or raw trace export. Add the eval extra to run benchmarks (see Evaluation).

The release includes agent presets for seven of the paper's eight backbones:

The paper's Qwen2.5-VL-7B-Instruct results, compared with training-based harnesses in its appendix, used a separate configuration that this release does not include.

The registry also covers every Qwen3-VL Instruct size (2B to 235B-A22B, with FP8 builds) and OpenAI's GPT-5 to GPT-6.1 models; to use one, copy its family's preset and change one line (other registered models).

To run all nine capabilities, prepare these services before starting the agent:

• Models: serve Qwen locally or connect to a GPT or Kimi endpoint, following the model setup guide. Every preset also needs a Qwen3-VL-8B-Instruct endpoint for search tools; the 8B preset can share it with the main model, while the 32B preset needs a second server for it. Sharing one server is the simplest deployment and scores within run-to-run noise of the paper setup; its cost is throughput, since search summarization and webpage extraction queue behind agent turns on the same GPU. To reproduce the paper exactly, or to keep agent throughput at higher concurrency, serve a second Qwen3-VL-8B instance and point AUXILIARY_MODEL_BASE_URL at it. The guide includes the tested model revision, serving versions, and GPU configuration.
• Perception: launch SAM 3, PaddleOCR-VL, and Depth Anything 3 using the tool service guide. These run as separate GPU services behind the router.
• Search: the three search tools call Serper, a paid Google Search API with a free starter quota: web/image/news search, Google Lens reverse image search, and page scraping for webpage visits. Configure its key using SERPER_API_KEY or SERPER_API_KEY_FILE. Reverse image search additionally needs a URL Serper can fetch for the query image: either the built-in image host behind a tunnel (no cloud account) or an Amazon S3 or Google Cloud Storage bucket, where the image is uploaded and handed to Serper as a signed URL. Follow the search and storage setup.

For the local Qwen setup, set these variables in both the router and agent terminals. Adjust the addresses to match your deployment:

export MODEL_BASE_URL=http://127.0.0.1:8222/v1
export MODEL_API_KEY=empty
export AUXILIARY_MODEL_BASE_URL=http://127.0.0.1:8222/v1
export AUXILIARY_MODEL_API_KEY=empty
export JUDGE_MODEL=Qwen/Qwen3-VL-8B-Instruct
export JUDGE_API_BASE=http://127.0.0.1:8222/v1/chat/completions
export JUDGE_API_KEY=empty
export TOOL_SERVER_URL=http://127.0.0.1:8101
export SAM3_SERVER_URL=http://127.0.0.1:8086
export OCR_SERVER_URL=http://127.0.0.1:8085
export DEPTH_ESTIMATION_SERVER_URL=http://127.0.0.1:8087

The empty keys above are for unauthenticated local endpoints. For GPT or Kimi, select its preset in the commands and Python example below, set MODEL_BASE_URL and MODEL_API_KEY for that provider, and keep the auxiliary connection pointing to Qwen. See the connection examples.

Start the router with the same configuration that the agent will use:

python -m openvlharness.services.server --config configs/agent/qwen3_8b_paper.yaml

In another terminal with the environment configured, check the router:

curl --fail http://127.0.0.1:8101/health
curl --fail http://127.0.0.1:8101/backend-status

These checks only show that the router is up, how many tools it registered, and its concurrency limits. Unreachable GPU workers, missing Serper keys and missing image hosting settings surface on the first call to the affected tool. /backend-status also reports degradations: counts of search and webpage-visit calls that fell back to raw provider content because the auxiliary model timed out or returned unusable output, and image searches whose published query image could not be fetched at its public URL (image_host), and Serper calls rejected because of the account (serper_account: credits exhausted, key invalid or payment required). A healthy router keeps these at zero; the same events are logged with the tokens SUMMARIZE_DEGRADED, EXTRACT_DEGRADED, IMAGE_HOST_UNREACHABLE and SERPER ACCOUNT-LEVEL FAILURE.

Then ask a question from Python:

from openvlharness.bootstrap import build_agent

agent = build_agent("configs/agent/qwen3_8b_paper.yaml")
result = agent.generate([
    {"type": "image", "value": "/path/to/image.jpg"},
    {"type": "text", "value": "How many bottles are visible?"},
])
print(result["prediction"])
trajectory = result["extra_records"]["trajectory"]

Replace the image path with your own image. Inputs can include multiple images; image values can be local paths, HTTP(S) URLs, or data URLs. The returned trajectory records the model's turns, tool calls, and evidence for inspection.

If result["prediction"] is Error: Unable to get response from the server., the agent produced no answer. When a model request failed, the cause is in result["extra_records"]["trajectory"]["error"]; without that key, the model ended without a final answer. Error: Reached the iteration limit without a final answer. means the question used all agent.max_iterations iterations; the trajectory's error records the limit.

Install the evaluation layer (benchmark loaders, judges and the runner) with the eval extra, keeping the model and tool services running:

python -m pip install -e '.[eval,qwen]'

This installs the dependencies of the openvlharness-eval command; python run_benchmark.py at the repository root is the same program. The evaluation code lives in evaluation_kit and needs no GPU packages.

Use the same connection variables as above. LMUData is required: benchmark data downloads there, and the evaluator raises an error if it is unset. Put it and the work directory on a fast local disk. This example evaluates 12 reproducibly sampled questions per dataset from the FSC147 and ODinW validation splits, using direct counting and box metrics without an LLM judge:

export LMUData=/path/to/local/datasets
openvlharness-eval \
  --config configs/agent/qwen3_8b_paper.yaml \
  --judge-config configs/judge_no_judge.yaml \
  --data FSC147_val ODinW13_bbox_val_qwen3 \
  --num-examples 12 --shuffle --seed 42 \
  --num-proc 1 --save-html all --work-dir /path/to/local/runs

--num-proc controls concurrent examples. Remove --num-examples and --shuffle to evaluate the full selected splits. For paper comparisons, also match the benchmark prompts, judge settings, and scoring protocol in Appendix D. The paper reports Qwen3-VL results averaged over three runs; a small subset checks the installation rather than reproducing those scores.

Dataset names passed to --data select loaders in evaluation_kit/dataset. The paper's 22 benchmarks (Table 1), the five additional search benchmarks of its appendix, and PixmoCount (supported but not reported in the paper) are:

Every dataset above except PixmoCount_val and CountQA downloads, with its images, from OpenVLHarness-Evaluation-Datasets into LMUData on first use; the loaders build those two from their original sources. PixmoCount_val is built from allenai/pixmo-count, with images fetched from their original URLs; images that can no longer be fetched are left out, so a fresh build can contain fewer than the 534 images the paper evaluated. CountQA is built from Jayant-Sravan/CountQA on the Hugging Face Hub. ODinW also downloads evaluation annotations into LMUData/GLIP unless GLIP_ROOT is set. The preparation scripts document how the released splits were built.

FVQA_test, Infoseek_GDM_test and EncyclopedicVQA_test are the largest search benchmarks (1,800, 2,000 and 1,960 questions). FVQA_test_300, Infoseek_GDM_test_300 and EncyclopedicVQA_test_300 are fixed 300-question subsets of them, a quick estimate of the full-split score at a fraction of the time and search cost; SimpleVQA_300 is the 300 English SimpleVQA questions WebWatcher evaluates on.

Benchmarks that need an LLM judge or answer extractor read it from JUDGE_MODEL, JUDGE_API_BASE (the full chat-completions URL) and JUDGE_API_KEY through configs/judge.yaml; any OpenAI-compatible endpoint works. The paper judged with Qwen3-VL-8B-Instruct, and with GPT-4o on the search and deep-research benchmarks:

export JUDGE_MODEL=gpt-4o
export JUDGE_API_BASE=https://api.openai.com/v1/chat/completions
export JUDGE_API_KEY=$OPENAI_API_KEY

One judge serves every dataset of an invocation, so evaluate the search benchmarks in a separate openvlharness-eval run from the others, or re-score a finished run with --mode eval --run-dir <run> --judge-config configs/judge.yaml after switching the variables.

Results are saved under <work-dir>/<model>/<agent>/<run_id>/<dataset>/. For image/text benchmarks, the main artifacts are:

Prediction filenames use the model ID's final component, such as Qwen3-VL-8B-Instruct.

The parent run directory contains run_manifest.json, agent_config.redacted.yaml, judge_config.redacted.yaml, and tool_env_snapshot.json, recording the effective configuration, resolved model profiles and service metadata. Add --resume to continue a compatible partial run; changes to the effective configuration require a new run. Rows that failed because a service was unreachable are scored as incorrect; once the service is back, rerun with --run-dir <run-dir> --ignore-failed to redo only those rows.

To evaluate saved predictions, select the run directly:

openvlharness-eval --mode eval --run-dir "$RUN_DIR" --data FSC147_val

Evaluation reads the saved model identity, prediction format and judge configuration; it does not load the inference agent, tokenizer or tool connections. Supply --judge-config when changing the judge or replacing redacted judge credentials. Inference resumes still validate the effective configuration and resolved model profiles. Without --run-dir, evaluation discovers the latest complete saved run under --work-dir; use --run-dir when several model/agent identities are present.

For Qwen3-VL-8B-only token-level export, add --save-raw-trace. This writes per-sample JSON and images under raw_traces/, with explicit error records for export failures. It requires a matching tokenizer and a server that returns token IDs; see raw trace export. Ordinary trajectory JSON and HTML reports work with every preset.

Where files go:

• Run outputs go under --work-dir (default ./outputs).
• Benchmark TSVs and images go in $LMUData.
• Image archives and ODinW annotations are downloaded into the Hugging Face cache (HF_HOME, default ~/.cache/huggingface) and then extracted into $LMUData, so they are stored twice: about 3.6 GB of archives for the datasets listed above. The CountQA and PixmoCount sources are cached there as well.
• ODinW annotations go in $LMUData/GLIP unless GLIP_ROOT is set.
• openvlharness-eval reads ./.env if present, without overriding variables that are already set. The router and the Python API do not read it.
• For inference, the default --config and --judge-config paths (configs/agent/qwen3_8b_paper.yaml and configs/judge.yaml) are relative, so run from the repository root or pass both.

Each agent YAML separates agent policy, memory, capabilities, and services. Start from a copy of a paper preset and select the capabilities your deployment can support.

GPU workers can run on another machine reachable from the router. The main Qwen model and the auxiliary Qwen summarizer also need model serving, separately from these perception workers; a hosted main model still needs auxiliary Qwen when the search capabilities are enabled. See the model and tool deployment guides.

Only reverse image search needs image hosting. Text search, including its image-search mode, and webpage visits do not. Configure SERPER_API_KEY or SERPER_API_KEY_FILE in the router environment, and follow the S3/GCS credential instructions when enabling image_search.

1. Remove the corresponding keys from capabilities.tools. For example, remove image_search to avoid the image-hosting requirement; remove text_search, image_search, and web_visit to disable Serper-backed tools altogether. Remove grounding, ocr, depth, and camera_pose to omit the GPU perception workers. Both depth and camera capabilities must be removed to stop needing DA3.
2. Delete the now-unused entries from services.tools. If all search capabilities are removed and nothing else uses it, also delete services.models.auxiliary. Every ${VARIABLE} remaining in the YAML is expanded during loading, and every configured model is validated, even if its capability is disabled. Removing unused connections lets you leave their environment variables unset.
3. Restart the router with the edited file and pass that same file to build_agent(...) or openvlharness-eval --config .... Check /tools and /llm-schema to confirm the selected tool set. Disabled capabilities are omitted from the model's tool schemas and their backend implementations are not loaded.

agent_name: MainAgent
agent_config_version: local_tools_v1
agent:
  model_service: orchestrator
  max_iterations: 15
memory:
  modules: [{name: image}, {name: tool_context}, {name: url}]
  emit_policy: always
capabilities:
  registries: [paper]
  tools:
    zoom:
      backend: zoom_in_image_by_bbox
    coding:
      generator: python_coding_agent
      executor: python_executor_local
services:
  client:
    base_url: ${TOOL_SERVER_URL}
  server:
    host: 127.0.0.1
    port: 8101
    registration: configured_only
  tools:
    zoom_in_image_by_bbox: {}
    python_coding_agent:
      model_service: orchestrator
      context: fresh
    python_executor_local:
      transport: local
      timeout_seconds: 30
  models:
    orchestrator:
      alias: gpt-6-luna
      base: ${MODEL_BASE_URL}
      key: ${MODEL_API_KEY}
      max_tokens: 16000
      reasoning_effort: medium

Registered models are listed in the model guide: copy the preset of the same family and change the orchestrator's alias. To add a model outside those families, register its served ID, API protocol, prompt, and supported options in model_registry.yaml. Auxiliary models can also use a registered compatible profile; multimodal Qwen remains the official search default.

For SAM 3, OCR, DA3, or GroundingDINO workers, point services.tools.<backend>.server_url to a service implementing that backend's HTTP contract. For example, merge this fragment into your configuration to use a SAM 3 worker on another host:

services:
  tools:
    sam3_segment:
      server_url: http://gpu-host:8086

Use the service base URL; the backend wrapper adds the request route. The service must accept the wrapper's request format and return the expected result fields. The optional GroundingDINO backend shows how to select a different implementation of the same grounding capability.

To integrate a service with a different API, add a wrapper under openvlharness/services/tools/<backend>/tool.py; the OCR wrapper is an HTTP example and the zoom backend is a small local example. Implement the backend's BaseTool.execute method, translate its response to the capability's expected fields, and register the backend as described below. Endpoints and credentials belong in deployment configuration or the service environment; public tool arguments describe the task.

New capability names and backends require source registration before they can be selected in YAML. The relevant files are:

1. Capability interface: create openvlharness/capabilities/<name>.py, following zoom.py. Define the public function schema, a typed output contract, and a Capability subclass of BaseCapability. Set name, schema, and backends (backend key to concrete tool class name). Implement normalize_result and process_output for canonical results and model-readable evidence. Use translate_args, validate_result, and store_result as needed to adapt arguments, validate results before registration, and retain numerical data for later tools. Image arguments such as image_id resolve through memory.
2. Backend implementation: add a package with __init__.py and tool.py under openvlharness/services/tools/<backend>/. Its tool class must match the capability's backends mapping. Implement BaseTool's execute method and return result data; the router adds the success/error envelope and tool-call ID.
3. Configuration registration: update PAPER_CAPABILITIES, PAPER_BACKENDS, and the backend/service-option validation in bootstrap.py. For another implementation of an existing capability, extend its backends mapping and accepted backend selections. Add HTTP workers to RAW_BACKENDS when they need server_url injected from their service configuration.
4. Router wiring: server.py imports the configured backend class. Extend its construction logic if your tool needs injected model clients or other dependencies. For resource-limited services, assign a BACKEND_GROUPS entry and a DEFAULT_LIMITS concurrency limit. Keep capability and backend registration available on both the agent and router hosts.
5. Select and verify: add capabilities.tools.<name>.backend and the matching services.tools.<backend> settings to your YAML. Add focused coverage in tests/test_capabilities.py and tests/test_services.py for valid calls, malformed arguments/results, and any stored image or numerical artifacts. Run a small question and inspect its trajectory to verify the model can use the evidence.

See runtime design for capability, memory, and service contracts.

openvlharness/          the harness (installed as the openvlharness package)
  agent/                orchestrator loop over model requests and tool calls
  capabilities/         the nine capability interfaces above
  memory/               per-question multimodal memory and Environment Update
  services/             model profiles, tool client and tool router
    tools/              tool backends (SAM 3, OCR, depth, search, Python executor, ...)
  prompts/              system prompts
  bootstrap.py          builds an agent from a YAML config
evaluation_kit/         benchmark loaders, judges and the runner (openvlharness-eval)
configs/                paper agent presets and judge settings
examples/               example agent configurations
docs/                   model, runtime and validation notes
tests/                  harness tests; tests/evaluation/ needs the eval extra
validation/             fixed perception subset for checking a tool deployment
assets/                 logo and README figures
run_benchmark.py        same program as openvlharness-eval

The harness never imports evaluation_kit; the evaluation layer uses the harness the same way any other application would.

To run the package tests:

python -m pip install -e '.[test,server]'
python -m pytest tests -q

The evaluator tests under tests/evaluation are collected only when the eval extra is installed as well.

The benchmark loaders and evaluators build on VLMEvalKit. The default tools use SAM 3, Depth Anything 3, PaddleOCR-VL and the Serper search APIs, and Qwen3-VL serves as the default backbone and search summarizer. We thank their authors for releasing them.

OpenVLHarness is released under the MIT License. The benchmark loaders and evaluators under evaluation_kit/ are derived from VLMEvalKit (Apache License 2.0), and the SAM 3 visualization helpers in openvlharness/services/tools/sam3_segment/viz/ are under the SAM License. See THIRD_PARTY_LICENSES.md for these notices and for the licenses of the models, services and datasets the default tools use, which are not covered by the MIT License.

If you use OpenVLHarness in your research, please cite:

@article{openvlharness2026,
  title   = {OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory},
  author  = {Zhou, Yu and Yang, Cheng-Fu and Sun, Rui and Wu, Di and Peng, Sibo and Zhang, Mingyang and Yang, Zhen and Zhang, Ruiyi and Peng, Nanyun and Gan, Zhe and Chang, Kai-Wei},
  journal = {arXiv preprint arXiv:2610.26860},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.26860}
}