VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion (CVPR 2026)

Official implementation of "VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion"

Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunyu Li, Hao Zhang, Han Xu, Jiayi Ma

---

🔥 News

• [2026] VideoFusion has been accepted to CVPR 2026.
• [2025] We release M3SVD, a large-scale aligned infrared-visible multi-modal video dataset for fusion & restoration.

---

🔎 Motivation

Most multi-modal fusion methods are designed for static images. Applying them frame-by-frame to videos often leads to:

• Temporal flickering (inconsistent fusion across frames)
• Under-utilization of motion/temporal cues

---

🧠 Architecture

The overall framework of our spatio-temporal collaborative video fusion network.

---

📦 M3SVD Dataset

• 220 temporally synchronized & spatially registered IR-VI videos
• 153,797 frames total
• Registered resolution 640×480, 30 FPS
• Diverse conditions: daytime / nighttime / challenging scenarios (e.g., occlusion, disguise, low illumination, overexposure)

Data Processing Workflow

Dataset Comparison (vs. prior works)

📌 Place dataset files following the dataloader requirement (see Dataset Preparation section).
🔗 Download links will be updated: (TBD)

---

⚙️ Installation

1) Clone

git clone git@github.com:Linfeng-Tang/VideoFusion.git
cd VideoFusion

2) Create Environment

conda create -n videofusion python=3.9 -y
conda activate videofusion
pip install -r requirements.txt

---

🚀 Quick Start (Testing)

Prepare

1. Download pretrained weights: (TBD)
2. Put weights into:

./pretrained_weights/

Run

python test.py -opt=./options/test/test_VideoFusion.yml

---

🚂 Training

1) Dataset Preparation

Download M3SVD and place it as:

<your_m3svd_root>/
  ├── train/
  │   ├── ir/seqxxx/*.png
  │   └── vi/seqxxx/*.png
  ├── val/
  │   ├── ir/...
  │   └── vi/...
  └── test/
      ├── ir/...
      └── vi/...

Then update options/train/train_VideoFusion.yml with the correct dataset root paths.

2) DDP Training

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port=7542 \
  train.py -opt ./options/train/train_VideoFusion.yml --launcher pytorch

---

🖼️ Qualitative Results

Fusion Quality (examples)

Qualitative comparison results on M3SVD and HDO datasets under degraded scenarios.

Quantitative comparison on the M3SVD and HDO datasets under degraded scenarios. Each video in M3SVD and HDO contains 200 and 150 frames, respectively. The best and second-best results are highlighted in Red and Purple, respectively.

Restoration / Robustness under Degradations

⏱️ Temporal Consistency

VideoFusion emphasizes temporal coherence. We provide temporal visualization examples:

Temporal variation of metrics on sequences.

Visual comparison of temporal consistency in source and fusion videos. Following DSTNet, we visualize pixels along selected columns (dotted line) and measure average brightness variation across frames.

📈 Ablation & Analysis

Ablation Study

---

🎯 Downstream / Tracking Demo

---

📝 Citation

If you find this work useful, please cite:

@inproceedings{Tang2026VideoFusion,
  title     = {VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion and Restoration},
  author    = {Tang, Linfeng and Wang, Yeda and Gong, Meiqi and Li, Zizhuo and Deng, Yuxin and Yi, Xunpeng and Li, Chunyu and Zhang, Hao and Xu, Han and Ma, Jiayi},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

---

❤️ Acknowledgments

This repository is built upon the excellent open-source framework BasicSR. We sincerely thank the authors for their great work and for making their code publicly available.

🤝 Contact

If you have any questions, please do not hesitate to contact linfeng0419@gmail.com.

---