OmniVid: A Generative Framework for Universal Video Understanding

Official PyTorch implementation of the following paper:

OmniVid: A Generative Framework for Universal Video Understanding, CVPR 2024.

Junke Wang^1,2, Dongdong Chen³, Chong Luo⁴, Bo He⁵, Lu Yuan³, Zuxuan Wu^1,2, Yu-Gang Jiang^1,2
1Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University
²Shanghai Collaborative Innovation Center of Intelligent Visual Computing
³Microsoft Cloud + AI, ⁴Microsoft Research Asia, ⁵University of Maryland, College Park

---

The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora.

Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding.

Setup

PLease download the datasets in the official websites. We adopt a unified format for different tasks, please refer to tools/generate_omni_json to see how to generate the corresponding annotations.

Training and Inferene

Please first build the environment following the script tools/prepare.sh. Then use the scripts under ./run_scripts/omnicaptioner to train and evaluate models. The pretrained checkpoints can be found in huggingface.

Visualizations

Citation

If you find this repository helpful, please consider citing:

@inproceedings{wang2024omnivid,
  title={OmniVid: A Generative Framework for Universal Video Understanding},
  author={Wang, Junke and Chen, Dongdong and Luo, Chong and He, Bo and Yuan, Lu and Wu, Zuxuan and Jiang, Yu-Gang},
  booktitle={CVPR},
  year={2024}
}