ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

📄 arXiv Paper | 🖥️ Github Code | 🤗 Data

Introduction

ProactiveVideoQA is the first comprehensive benchmark designed to evaluate a system's ability to engage in proactive interaction in multimodal dialogue settings. Unlike traditional turn-by-turn dialogue systems, in proactive intraction model need to determine when to repsond during the playback, so both response timing and response textual content are important points for evaluation.

This repository hosts the dataset for ProactiveVideoQA, along with the evaluation code of our proposed temporal-aware metric, PAUC (Proactive Area Under the Curve).

Dataset Statistics

ProactiveVideoQA contains 4 tasks:

1. Proactive web-video QA [WEB]: centering on general web-video understanding.

2. Proactive ego-centric video QA [EGO]: centering on first-person-view video comprehension, particularly relevant in robotics and daily assistant applications.

3. Proactive TV-series video QA [TV]: emphasizing dialogue and social relationship understanding with speech input, and

4. Proactive video anomaly detection [VAD] targeting surveillance video monitoring and alerting.

• 1377 videos from different sources
• 1427 different qeustions, and 3510 ground truth reply turns
• Fully proactive questions and open-ended answers ✅

Project Structure

ProactiveVideoQA/  
├── data/       # ProactiveVideoQA test data (Download from [🤗Huggingface](https://huggingface.co/datasets/wangyueqian/ProactiveVideoQA))
├── pauc/       # PAUC scoring scripts  

Dataset

Download the annotations and videos from 🤗Huggingface, and rename the folder to ./data.

Evaluation code

See pauc/README.md.

Contact

If you find ProactiveVideoQA useful in your research, please cite our paper:

@misc{wang2025proactivevideoqacomprehensivebenchmarkevaluating,
      title={ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models}, 
      author={Yueqian Wang and Xiaojun Meng and Yifan Wang and Huishuai Zhang and Dongyan Zhao},
      year={2025},
      eprint={2507.09313},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.09313}, 
}