Colab-Zooming-Slow-Mo (CVPR-2020)

By Xiaoyu Xiang^*, Yapeng Tian^*, Yulun Zhang, Yun Fu, Jan P. Allebach⁺, Chenliang Xu⁺ (^* equal contributions, ⁺ equal advising)

This is a Google Colab Notebook for the official Pytorch implementation of Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution.

Simply download Colab-Zooming-Slow-Mo.ipynb and open it inside your Google Drive or click here and copy the file with "File > Save a copy to Drive..." into your Google Drive.

Important information

• If you can't open Colab-Zooming-Slow-Mo.ipynb inside your Google Drive, try this colab link and save it to your Google Drive. The "open in Colab"-button can be missing in Google Drive, if that person never used Colab.
• Google Colab does assign a random GPU. It depends on luck.
• The Google Colab VM does have a maximum session length of 12 hours. Additionally there is a 30 minute timeout if you leave colab. The VM will be deleted after these timeouts.

Paper | Demo (YouTube) | 1-min teaser (YouTube) | 1-min teaser (Bilibili)

Input-> <- Output

Updates

• Our paper will have Q&A session on June 16, 2-4 am/pm PDT. Welcome to come and ask questions!
• 2020.3.13 Add meta-info of datasets used in this paper
• 2020.3.11 Add new function: video converter
• 2020.3.10: Upload the complete code and pretrained models

Introduction

The repository contains the entire project (including all the preprocessing) for one-stage space-time video super-resolution with Zooming Slow-Mo.

Zooming Slow-Mo is a recently proposed joint video frame interpolation (VFI) and video super-resolution (VSR) method, which directly synthesizes an HR slow-motion video from an LFR, LR video. It is going to be published in CVPR 2020. The most up-to-date paper with supplementary materials can be found at arXiv.

In Zooming Slow-Mo, we firstly temporally interpolate features of the missing LR frame by the proposed feature temporal interpolation network. Then, we propose a deformable ConvLSTM to align and aggregate temporal information simultaneously. Finally, a deep reconstruction network is adopted to predict HR slow-motion video frames. If our proposed architectures also help your research, please consider citing our paper.

Zooming Slow-Mo achieves state-of-the-art performance by PSNR and SSIM in Vid4, Vimeo test sets.

Citations

If you find the code helpful in your resarch or work, please cite the following papers.

@InProceedings{xiang2020zooming,
  author = {Xiang, Xiaoyu and Tian, Yapeng and Zhang, Yulun and Fu, Yun and Allebach, Jan P. and Xu, Chenliang},
  title = {Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={3370--3379},
  month = {June},
  year = {2020}
}

@InProceedings{tian2018tdan,
  author={Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu},
  title={TDAN: Temporally Deformable Alignment Network for Video Super-Resolution},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={3360--3369},
  month = {June},
  year = {2020}
}

@InProceedings{wang2019edvr,
  author    = {Wang, Xintao and Chan, Kelvin C.K. and Yu, Ke and Dong, Chao and Loy, Chen Change},
  title     = {EDVR: Video restoration with enhanced deformable convolutional networks},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  month     = {June},
  year      = {2019},
}

Contact

Xiaoyu Xiang and Yapeng Tian.

You can also leave your questions as issues in the repository. We will be glad to answer them.

License

This project is released under the GNU General Public License v3.0.

Acknowledgments

Our code is inspired by TDAN-VSR and EDVR.