A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
Accepted to ECCV 2026 (Poster)
Important
This repository has moved. It now hosts the project page only, and contains no code.
The official, up-to-date home of TransVLM is:
| Official code repository | https://github.com/heygen-com/TransVLM |
| Official project page | https://heygen-com.github.io/TransVLM/ |
| Model weights | https://huggingface.co/HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct |
Go there for the model weights, the inference and evaluation code, and the issue tracker. Issues and pull requests opened here will not be tracked.
This repository keeps serving the project page at https://chence17.github.io/TransVLM/ so that the URL printed in the paper keeps working. That copy is a mirror of the official page above.
- TransVLM was accepted to ECCV 2026 as a Poster.
- Model weights, inference code and evaluation code are released at
heygen-com/TransVLM. See Release Progress. - The official code repository moved to
heygen-com/TransVLM.
Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots. We address this fundamental limitation by formalizing the Shot Transition Detection (STD) task. Rather than searching for ambiguous points, STD explicitly detects the continuous temporal segments of transitions. To tackle this, we propose TransVLM, a Vision-Language Model (VLM) framework for STD. Unlike regular VLMs that predominantly rely on spatial semantics and struggle with fine-grained inter-shot dynamics, our method explicitly injects optical flow as a critical motion prior at the input stage. Through a simple yet effective feature-fusion strategy, TransVLM directly processes concatenated color and motion representations, significantly enhancing its temporal awareness without incurring any additional visual token overhead on the language backbone. To overcome the severe class imbalance in public data, we design a scalable data engine to synthesize diverse transition videos for robust training, alongside a comprehensive benchmark for STD. Extensive experiments demonstrate that TransVLM achieves superior overall performance, outperforming traditional heuristic methods, specialized spatiotemporal networks, and top-tier VLMs.
Qualitative results and the comparison against baselines are on the official project page.
- Model weights
- Inference code
- Evaluation code
- Data engine code
- STD benchmark data
- Re-annotated dataset labels
- Leaderboard
🚧 The remaining items are being prepared for release.
The code lives in the official repository. You need Python 3.12, a CUDA GPU, and ffmpeg on
PATH. Clone and install first: setup has a few sharp edges (a cuDNN version floor, mutually
exclusive backend venvs, and FFmpeg library discovery), so follow that repository's
inference/README.md rather than guessing:
git clone https://github.com/heygen-com/TransVLM
cd TransVLM/inference
uv venv && source .venv/bin/activate
uv sync --group cu130 --group dev
uv pip install nvidia-cudnn-cu13==9.16.0.29 # required, and every uv sync undoes itThen fetch the checkpoint and run detection on one video:
hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1
python infer_video.py \
--video /path/to/video.mp4 \
--ckpt-dir ./pretrained/TransVLM-v1 \
--output-jsonl out.jsonlImportant
This checkpoint takes 6-channel RGB + optical-flow input, so a plain transformers
Qwen3-VL pipeline will not produce correct results. Use the released inference code.
Every option, the output format and the three backends are documented here:
| Topic | Document |
|---|---|
| Inference | https://github.com/heygen-com/TransVLM/blob/main/inference/README.md |
| Evaluation | https://github.com/heygen-com/TransVLM/blob/main/evaluation/README.md |
| Path | Contents |
|---|---|
docs/ |
Project page sources (figures, videos, interactive comparison widgets) |
assets/ |
Logos used by this README |
Code is not mirrored here. See heygen-com/TransVLM.
@inproceedings{chen2026transvlm,
title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
booktitle={European Conference on Computer Vision},
year={2026},
organization={Springer}
}The STD benchmark is intended as a living resource. We are actively exploring smaller and more efficient VLM backbones for the STD task, and the resulting checkpoints, training recipes, and evaluation results will be open-sourced at heygen-com/TransVLM as they become available.
We welcome community submissions of smaller-model baselines on the STD benchmark. If you have a more compact or more efficient method for STD, please open an issue or pull request on the official repository with: (i) the model description and total parameter count; (ii) per-method evaluation results following the format on the official project page; and (iii) reproduction instructions. Approved submissions will be added to the leaderboard on the project page.
TransVLM builds on Qwen3-VL-4B-Instruct and uses NeuFlow v2 for optical-flow estimation. We thank both teams for releasing their work.
Released under the Apache License 2.0. Vendored third-party components and their
upstream attribution are listed in the
NOTICE file of the official
repository. The released weights derive from Qwen3-VL-4B-Instruct and are distributed under
Apache-2.0 as well.