A challenging benchmark for evaluating multi-evidence integration capabilities of vision-language models
🎉 HERBench has been accepted to CVPR 2026!
🆕 New: Lite-v2 (refined). A refreshed Lite split is now available on Hugging Face as the
lite_v2config (1,971 questions / 68 videos), in which 9 of the 12 tasks were regenerated and went through additional manual refinement for higher quality. TSO, SVA, and RLPC are carried over unchanged fromlite. Load it withload_dataset("DanBenAmi/HERBench", "lite_v2"); the originalliteconfig remains available unchanged.
HERBench is a benchmark designed to evaluate how vision-language models integrate and reason over multiple pieces of evidence in videos. Unlike existing benchmarks that often allow questions to be answered from a single salient cue, HERBench enforces a High Evidential Requirement (ER): each question requires aggregating at least k ≥ 3 distinct, temporally separated visual cues.
Key Features:
- 27,936 five-way multiple-choice questions across 12 compositional tasks
- 335 unique videos from diverse sources
- Average video length: 395 seconds (6.6 minutes)
- Mean MRFS (Minimum Required Frame-Set): ~5.5 - significantly higher than existing benchmarks
- Rigorous quality control to eliminate language priors and single-frame shortcuts
- Available in Full (~161 GB) and Lite (~35 GB) versions
| Metric | Full Version | Lite Version |
|---|---|---|
| 📊 Total Questions | 27,936 five-way multiple-choice | 5,960 questions (21.3%) |
| 🎬 Videos | 335 unique videos | 68 unique videos (20.3%) |
| ⏱️ Avg. Video Length | 424 seconds | 421 seconds |
| 📁 Total Size | ~161 GB | ~35 GB |
Current video QA benchmarks often allow models to answer questions using single frames or limited context, failing to test true multi-evidence reasoning. HERBench addresses this by:
✅ Enforcing multi-evidence integration - Each question requires k ≥ 3 temporally separated frames ✅ Preventing single-frame shortcuts - Questions cannot be answered from isolated frames ✅ Testing compositional reasoning - Combines temporal, spatial, and causal reasoning ✅ Evaluating long-video understanding - Average video length of 6.6 minutes
HERBench is available in two versions to accommodate different storage and computational constraints:
- 27,936 questions across 335 videos
- Complete benchmark for comprehensive evaluation
- Recommended for: Final paper results, thorough model evaluation, benchmarking
- 5,960 questions across 68 videos (21.3% subset)
- Same task distribution and difficulty as full version
- Videos sampled to maintain diversity across all 12 tasks
- Recommended for: Quick prototyping, limited storage, initial experiments, development
Both versions maintain the same quality standards and high evidential requirements!
HERBench is distributed exclusively through HuggingFace. The dataset is available at huggingface.co/datasets/DanBenAmi/HERBench.
The easiest way to use HERBench is to load it directly from HuggingFace in your code:
from datasets import load_dataset
# Load FULL version (27,936 questions)
dataset_full = load_dataset("DanBenAmi/HERBench", "full")
print(f"Total questions: {len(dataset_full['test'])}")
# Load LITE version (5,960 questions)
dataset_lite = load_dataset("DanBenAmi/HERBench", "lite")
print(f"Lite questions: {len(dataset_lite['test'])}")
# Load LITE-v2 (refined) - 1,971 questions, same 68 videos
dataset_lite_v2 = load_dataset("DanBenAmi/HERBench", "lite_v2")
print(f"Lite-v2 questions: {len(dataset_lite_v2['test'])}")
# Access a sample
sample = dataset_full['test'][0]
print(f"Question: {sample['question']}")
print(f"Choices: {sample['choices']}")
print(f"Answer: {sample['answer']}")
print(f"Video: {sample['video_path']}")This repository's evaluation code is pre-configured to load from HuggingFace by default (see configs/config.yaml).
If you need the video files locally (e.g., for custom processing), download them using the HuggingFace CLI:
# Install dependencies
pip install huggingface-hub
# Download FULL version (27,936 questions, ~161 GB)
huggingface-cli download DanBenAmi/HERBench --repo-type dataset --local-dir HERBench
# Download LITE version only (5,960 questions, ~35 GB)
huggingface-cli download DanBenAmi/HERBench \
--include "data/herbench_lite.parquet" \
--include "data/*metadata.json" \
--include "videos/videos.tar.part.00" \
--include "videos/videos.tar.part.01" \
--include "videos/videos.tar.part.02" \
--include "videos/videos.tar.part.03" \
--include "videos/videos_lite_info.txt" \
--include "videos/videos.tar.checksums.txt" \
--local-dir HERBenchcd HERBench/videos
cat videos.tar.part.* > videos_full.tar
tar -xvf videos_full.tar
rm videos_full.tar # Clean upcd HERBench/videos
cat videos.tar.part.{00..03} > videos_lite.tar
tar -xvf videos_lite.tar
rm videos_lite.tar # Clean up- Python 3.8-3.12
- CUDA-compatible GPU (recommended: 48GB+ VRAM for large models)
- PyTorch 2.0+
# Clone the repository
git clone https://github.com/DanBenAmi/HERBench.git
cd HERBench
# Create and activate conda environment
conda env create -f environment.yml
conda activate herbench
# Install HERBench package
pip install -e .Currently implemented:
- Qwen2.5-VL-7B-Instruct (
model=qwen25vl) - InternVL3.5-8B (
model=internvl35)
The evaluation code automatically loads the dataset from HuggingFace (configured in configs/config.yaml):
# Evaluate on FULL version (27,936 questions) - default
python evaluation/run_evaluation.py model=qwen25vl frame_selector=uniform
# Evaluate on LITE version (5,960 questions) - faster prototyping
python evaluation/run_evaluation.py model=qwen25vl frame_selector=uniform data.hf_config=lite
# InternVL3.5-8B with BLIP-based frame selection
python evaluation/run_evaluation.py model=internvl35 frame_selector=blip
# Use local videos if you've downloaded them
python evaluation/run_evaluation.py model=qwen25vl data.videos_dir=/path/to/videosNote: The first run will download the dataset from HuggingFace. Videos are streamed directly, so local download is optional.
# Calculate accuracy
python evaluation/calculate_accuracy.py \
--predictions results/predictions_qwen25vl_uniform.json
# Calculate MRFS (Minimum Required Frame Set)
python evaluation/calculate_mrfs.py \
model=qwen25vl \
frame_selector=blip \
mrfs.min_frames=1 \
mrfs.max_frames=16For detailed configuration options, advanced usage, and information on adding custom models or frame selectors, see USAGE_GUIDE.md.
State-of-the-art models achieve only 31-42% accuracy (random baseline: 20%), revealing fundamental limitations in multi-evidence reasoning:
HERBench requires significantly more evidence integration than existing benchmarks:
Key Findings:
- Frame selection is a major bottleneck - Adaptive selectors outperform uniform sampling but still lag behind oracle keyframes
- Multi-evidence reasoning is a bottleneck - Even with oracle frames, models struggle to integrate complementary information
Each question in HERBench is designed to require:
- Multiple evidence pieces (k ≥ 3 frames minimum)
- Temporal separation between evidence frames
- Compositional reasoning across evidence
- Integration of visual information from different moments
| Task Name | Abilities Tested | Example |
|---|---|---|
| [TSO] Temporal Shot Ordering | Understanding event order, high-level scene transitions, chronological reconstruction using content cues | "The following 4 shots take place in the video: [Shot 1-4 descriptions]. Select the option that correctly reflects the order in which these shots occur in the video." |
| [MPDR] Multi-Person Duration Reasoning | Fine-grained time-span contrasts, interval statistics, comparing appearance durations across individuals | "These people were in the video: [Person 1-3 descriptions]. Who stayed in the frame FOV for the longest time?" |
| [ASII] Action Sequence Integrity & Identification | Micro-level task sequencing, action ordering, temporal understanding of fine-grained activities | "What is the correct temporal order of the 5 narrated events? (e.g., 1. slide coffee capsule -> 2. close lid -> 3. turn off processor -> 4. place orange -> 5. put down sponge)" |
| Task Name | Abilities Tested | Example |
|---|---|---|
| [AGBI] Appearance-Grounded Behavior Interactions | Social and relational cues, identity maintenance across time, interaction recognition | "In the video there is exactly one individual that fits the following description: [Appearance]. Who is accompanying the person as they walk across the frame?" |
| [AGAR] Appearance-Grounded Attribute Recognition | Moment-specific attribute extraction, target tracking, reading contextual details from specific individuals | "In the video there is exactly one individual that fits the following description: [Appearance]. What color is the jacket worn by the individual who remains seated as the main subject walks past?" |
| [AGLT] Appearance-Grounded Localization Trajectory | Global path-level motion reasoning, trajectory tracking, spatial exit/entry point identification | "In the video there is exactly one individual that fits the following description: [Appearance]. How does the person exit the frame at the end of their path?" |
| Task Name | Abilities Tested | Example |
|---|---|---|
| [FAM] False Action Memory | Action-level absence detection, exhaustive video-wide verification, distinguishing what did not occur | "Which of the following actions did NOT occur in the video? (A) open drawer (B) open up fridge (C) turn on tap..." |
| [SVA] Scene Verification Arrangement | Shot-level fidelity checking, chronology verification, distinguishing real from fabricated descriptions | "From the correctly described shots, which is the one that appears first in the video? [Multiple shot descriptions provided]" |
| [FOM] False Object Memory | Object-level absence detection, interaction verification, identifying non-interacted objects | "Which object did the camera wearer NOT interact with? (A) Cutting board (B) Sponge (C) Dish soap (D) Garlic presser..." |
| Task Name | Abilities Tested | Example |
|---|---|---|
| [MEGL] Multi-Entities Grounding & Localization | Set membership verification, identity deduplication, exact-match appearance verification | "Which of the following people appeared in the video (the person description must match exactly): [Person 1-3 descriptions] - A) only 1 and 3" |
| [AC] Action Counting | Event-accumulation across dispersed moments, counting repeated actions, temporal aggregation | "How many times does the action-object pair 'close tap' occur? A) 3 B) 5 C) 7..." |
| [RLPC] Region-Localized People Counting | Region-conditioned identity aggregation, spatial partitioning, counting with spatial constraints | "How many people entered the frame through the top edge? Select the range that includes the correct count." |
Videos are sourced from diverse, high-quality datasets:
- WildTrack (56 segments): Multi-camera pedestrian tracking scenes
- HD-EPIC (176 videos): First-person egocentric daily activities
- PersonPath22 (24 videos): Person tracking scenarios
- Movie Trailers (81 videos): Narrative storytelling content
If you use HERBench in your research, please cite:
@article{herbench2025,
title={HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering},
author={Ben-Ami, Dan and Serussi, Gabriele and Cohen, Kobi and Baskin, Chaim},
journal={arXiv preprint arXiv:2512.14870},
year={2025}
}This dataset is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).
Research Use Only. HERBench is released strictly for non-commercial research and educational purposes. The benchmark is constructed using videos originating from existing datasets and platforms, including WildTrack, HD-EPIC, PersonPath22, and publicly available online videos (e.g., YouTube trailers). All rights to the original video content remain with their respective owners and licensors.
HERBench does not claim ownership of any underlying video content. The use of such materials is intended solely for academic evaluation and analysis, in accordance with the terms of the respective source datasets and platforms.
Removal upon request. If any content owner or rights holder believes that their material has been included in HERBench in a manner that violates applicable terms or rights, please contact us. Upon notification, we will promptly investigate the request and remove the relevant content as appropriate.
We thank the creators of the original video datasets (WildTrack, HD-EPIC, PersonPath22) for making their data publicly available. We also acknowledge the movie studios for releasing promotional trailers.
- Dan Ben-Ami - danbenami3@gmail.com
- Gabriele Serussi - serussigabriele@gmail.com
- Kobi Cohen
- Chaim Baskin
- Issues: GitHub Issues
- Email: danbenami3@gmail.com
- 📄 Paper: https://arxiv.org/abs/2512.14870
- 💻 Code: https://github.com/DanBenAmi/HERBench
- 🌐 Project Page: https://gabrieleserussi.github.io/HERBench/
- 🤗 Dataset: https://huggingface.co/datasets/DanBenAmi/HERBench
Built with care for advancing video understanding research
If you find HERBench useful, please ⭐ star our repository!