Unified Multimodal Deep Research Agent for Image and Video

Unified Tool Set  ·  VGEG-Centered Data Engine  ·  Rubric Reward  ·  Multi-Image / Video Benchmarks

• 📖 Introduction
• 🗺️ Overview
• 🍭 Method Overview
• 📊 OneSearch Benchmarks
• 🏆 Main Results
• 🔎 Case Study
• 🤗 Models & Datasets
• 📁 Repository Layout
• 🛠️ Getting Started
• 🏋️ Agentic SFT
• 🚀 Agentic RL
• 🔬 Inference & Evaluation
• 🙌 Acknowledgements
• 📄 Citation
• ⚖️ License

OneSearch-VL is a unified multimodal deep research agent for single images, multi-image collections, and videos. It locates visual clues, retrieves external knowledge, and combines evidence into supported answers.

At its core is the Visually Grounded Evidence Graph (VGEG), which connects visual anchors, source-supported facts, and answer-producing operations. This shared structure supports three parts of our approach:

• Data construction: a VGEG-centered data engine generates and verifies research questions and expert tool-use trajectories.
• Evidence supervision: the Evidence-aware Visual-Grounded Rubric reward (EVGR) supervises evidence traceability and visual grounding during RL.
• Evaluation: OneSearch-MI-Bench and OneSearch-Video-Bench evaluate how agents retrieve and compose evidence across six research operations.

Starting from Qwen3-VL-8B, OneSearch-VL learns a shared research policy through joint SFT and RL across all three visual input types. The agent combines visual inspection, image and text search, and video frame selection in a common tool environment. VGEGs provide supervision and evaluation references; the agent receives the visual input, question, and tools and gathers its own evidence.

This repository provides SFT, RL, and inference and evaluation code. Model weights and data are available under OneSearch-VL on Hugging Face.

Our data engine discovers visual anchors, connects them to verified web facts, and constructs compositional questions with explicit evidence dependencies. Expert trajectories are filtered for answer correctness and process quality. Together with adapted single-image data from OpenSearch-VL, this produces OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K for unified training.

VGEG-centered data construction, from visual sources to verified research tasks and expert trajectories.

EVGR uses VGEG-derived rubrics to evaluate whether a rollout obtains the required facts (evidence traceability) and uses the correct images, regions, or video frames (visual grounding). Combined with answer and query rewards, it encourages the agent to ground its answers in the evidence it actually gathers.

EVGR supervises evidence traceability and visual grounding during reinforcement learning.

OneSearch-MI-Bench contains 301 multi-image questions, and OneSearch-Video-Bench contains 307 video questions. Both organize evaluation around six research operations: single-anchor lookup, multi-hop retrieval, knowledge-conditioned count, multi-anchor join, arithmetic, and comparison.

The OneSearch benchmarks cover six operations for retrieving and composing visual and web evidence.

Multi-image tasks use unordered image sets with stable visual references; video tasks retain temporal structure. The dataset cards provide media downloads and preparation instructions.

• Single-image research: OneSearch-VL-8B achieves 58.3 average accuracy across seven benchmarks, improving over OpenSearch-VL-8B by 1.7 points.
• Multi-image and video research: compared with Qwen3-VL-8B with tools, it improves accuracy by 20.2 points on OneSearch-MI-Bench, 17.6 points on OneSearch-Video-Bench, and 27.0 points on VideoDR.
• Ablations: joint SFT achieves the best average in the data-mixture ablation. Full EVGR adds 3.8 points over answer and query rewards on the paper's six-benchmark ablation average.

All gains are in percentage points. See the paper for detailed comparisons.

In this multi-image example, the agent identifies Ford and Brembo from separate visual anchors, retrieves their founding years, and answers that the manufacturers were founded 58 years apart.

Multi-image arithmetic: visual grounding, external fact retrieval, and answer composition.

• OneSearch-VL-8B: our unified agent for single-image, multi-image, and video deep research.
• OneSearch-VL-SFT-110K: approximately 110K expert trajectories for joint supervised fine-tuning.
• OneSearch-VL-RL-10K: approximately 10K tasks for reinforcement learning.

For setup, follow the SFT data preparation guide and RL registration guide. Benchmark downloads are linked in OneSearch Benchmarks.

OneSearch-VL/
├── images/                         # README assets and project logo
├── SFT/                            # Joint agentic supervised fine-tuning
│   ├── examples/onesearch/          # Qwen3-VL-8B SFT-110K recipe
│   ├── data/dataset_info.json       # Three modality dataset registrations
│   ├── scripts/relocate_media.py    # Resolve downloaded media paths
│   ├── run_train.sh
│   └── README.md
├── RL/                             # Asynchronous agentic reinforcement learning
│   ├── rllm/                       # rLLM, verl, rollout workflow, and launchers
│   ├── Megatron-LM/                 # Policy training backend
│   ├── mbridge/                     # Hugging Face ↔ Megatron bridge
│   ├── register_dataset.py
│   └── README.md
├── onesearch_vl/                    # Shared visual tools, inference, and evaluation
│   ├── onesearch_infer/             # Prompts, tools, model backends, and agent loop
│   ├── run_infer.py                 # Unified modality entry point
│   ├── run_eval.sh                  # Benchmark evaluation
│   └── README.md
├── scripts/                         # Source verification and release export
├── THIRD_PARTY_NOTICES.md
├── LICENSE
└── README.md

git clone https://github.com/appletea233/OneSearch-VL.git
cd OneSearch-VL

Run the commands below from the repository root. Paths such as /data/... and /models/... are placeholders for your local datasets and checkpoints. Use a separate environment for each component; installation commands and backend requirements are documented in the linked guides.

Copy the templates for the components you use, then fill in your model, retrieval, judge, and media-publication settings:

cp RL/.env.example RL/.env
cp onesearch_vl/.env.example onesearch_vl/.env

RL launchers load RL/.env; inference and evaluation shell wrappers load onesearch_vl/.env. Both accept ENV_FILE to select a different file. When running inference Python entry points directly, export the settings first:

set -a
source onesearch_vl/.env
set +a

See the service configuration guide for the required providers and credentials. Local .env files are ignored by Git.

Download SFT-110K, extract its media archives into their modality directories, then prepare the registered training files and launch:

python SFT/scripts/relocate_media.py \
  --base-dir /data/OneSearch-VL-SFT-110K \
  --output-dir SFT/data

bash SFT/run_train.sh \
  model_name_or_path=/models/Qwen3-VL-8B-Instruct \
  output_dir=/checkpoints/onesearch-sft

The recipe uses learning rate 1e-5, eight epochs, and DeepSpeed ZeRO-2. The vision tower and multimodal projector remain frozen.

Dataset layout, installation, training parameters, and multi-node setup: SFT/README.md.

Download the prepared RL-10K bundle, register the train split, and launch from your SFT checkpoint:

DATA_ROOT=/data/OneSearch-VL-RL-10K \
  bash RL/rllm/vision_deepresearch_async_workflow/data_prepare/register_rl_dataset.sh \
  --media-root /data/OneSearch-VL-RL-10K

# Set MODEL_PATH, OUTPUT_DIR, FORMAL_EVGR_DATA_ROOT, JUDGE_API_KEY,
# SEED_API_BASE, and SEED_API_KEY in RL/.env.
# For the path above, set FORMAL_EVGR_DATA_ROOT to the same data directory.
bash RL/rllm/vision_deepresearch_async_workflow/run/qwen3-vl-8b-single-node.sh

The released launcher uses the RLOO advantage estimator, eight responses per prompt, prompt batch 256, minibatch 64, and 201 training steps. Single-node and multi-node launchers default to 1 × 8 and 4 × 8 GPUs respectively. The released recipe is train-only.

Installation, dataset registration, Ray setup, rewards, and multi-node launch: RL/README.md.

Use the published OneSearch-VL-8B checkpoint with the local 8b backend. Install the inference dependencies and export your service settings as described in Getting Started. For the OneSearch benchmarks, run the dataset's prepare.py first to generate its runtime/ inputs.

# Single image: supply a supported parquet benchmark.
python onesearch_vl/run_infer.py --modality single_image \
  --model 8b --checkpoint OneSearch-VL/OneSearch-VL-8B \
  --data-path /data/single_image.parquet --output-dir ./outputs/single_image

# OneSearch-MI: preserve the image-index mapping in the prepared manifest.
python onesearch_vl/run_infer.py --modality multi_image \
  --model 8b --checkpoint OneSearch-VL/OneSearch-VL-8B \
  --csv /data/OneSearch-MI-Bench/runtime/mi.csv \
  --manifest /data/OneSearch-MI-Bench/runtime/mi_manifest.jsonl \
  --output-dir ./outputs/onesearch_mi

# OneSearch-Video: use the prepared video and original-resolution frame manifest.
python onesearch_vl/run_infer.py --modality video \
  --model 8b --checkpoint OneSearch-VL/OneSearch-VL-8B \
  --csv /data/OneSearch-Video-Bench/runtime/video.csv \
  --manifest /data/OneSearch-Video-Bench/runtime/video_manifest.jsonl \
  --output-dir ./outputs/onesearch_video

For an HTTP model server, use --model vllm and configure VLLM_API_BASE and VLLM_MODEL instead of passing a local checkpoint. The public Claude Messages API is also supported; see model backends.

Evaluate the saved trajectories with the corresponding benchmark inputs:

bash onesearch_vl/run_eval.sh image \
  --traj_dir ./outputs/onesearch_mi \
  --csv /data/OneSearch-MI-Bench/runtime/mi.csv

bash onesearch_vl/run_eval.sh video \
  --traj_dir ./outputs/onesearch_video \
  --csv /data/OneSearch-Video-Bench/runtime/video.csv

Single-image and multi-image evaluation default to the onesearch protocol; OneSearch-Video and VideoDR default to videodr. Both use JUDGE_MODEL (default: gpt-4o).

Input schemas, video preparation, model backends, and judge protocols: onesearch_vl/README.md.

This project builds on OpenSearch-VL, LLaMA-Factory, rLLM, verl, Megatron-LM, mbridge, sglang, and Qwen-VL. The bundled frameworks include project modifications used by the training and inference workflows.

We also thank the authors of Search-R1 and Vision-DeepResearch whose ideas inspired our multi-turn search-augmented RL formulation.

OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video.

@article{li2026onesearch,
  title={OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video},
  author={Li, Hongyu and Zhang, Manyuan and Feng, Kaituo and Chen, Shu and Zheng, Dian and Li, Hao and Yu, Hao and Chen, Zhangquan and Guo, Zoey and Zhang, Ray and others},
  journal={arXiv preprint arXiv:2610.12419},
  year={2026}
}

This repository is released under the Apache License 2.0. See THIRD_PARTY_NOTICES.md for bundled components, which retain their licenses and copyright notices.