MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance

AAAI 2024

Ernie Chu, Tzuhsuan Huang, Shuo-Yen Lin, Jun-Cheng Chen

Research Center for Information Technology Innovation, Academia Sinica

Project Page | Paper | arXiv | Colab

Colab demo

• Ground-truth flow
• Predicted flow

Environment setup

We use conda to maintain the Python environment

conda env create -f environment.yml

The implementation of MeDM is in pipeline_medm.py. We incorporate MeDM into the snapshot of Diffusers at version 0.20.0. To install it, simply use

cd diffusers-0.20.0
pip install .

Minimal examples

We provide two simple examples on how to use MeDMPipeline in colabs. Remember to skip the colab-specific blocks when running locally.

Citation

If you find our work useful, please consider cite this work as

@inproceedings{chu2024medm,
      title={MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance},
      author={Ernie Chu and Tzuhsuan Huang and Shuo-Yen Lin and Jun-Cheng Chen},
      booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
      year={2024}
}