PyTorch implementation of Super SloMo by Jiang et al.
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Update issue templates Jan 15, 2019
data Added custom dataset creator Dec 28, 2018
misc update readme. Dec 27, 2018
.gitignore Added video converter script. Dec 30, 2018
LICENSE Initial commit Dec 25, 2018
README.md Added progress bar. Jan 5, 2019
dataloader.py Fixed issue #10 Jan 3, 2019
model.py accomodating -> accommodating Dec 29, 2018
train.ipynb Fixed issue #6 Dec 31, 2018
train.py Updated README.md Jan 3, 2019
video_to_slomo.py Added progress bar. Jan 5, 2019

README.md

Super-SloMo MIT Licence

PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun D., Jampani V., Yang M., Learned-Miller E. and Kautz J. [Project] [Paper]

Results

Results on UCF101 dataset using the evaluation script provided by paper's author. The get_results_bug_fixed.sh script was used. It uses motions masks when calculating PSNR, SSIM and IE.

Method PSNR SSIM IE
DVF 29.37 0.861 16.37
SepConv - L_1 30.18 0.875 15.54
SepConv - L_F 30.03 0.869 15.78
SuperSloMo_Adobe240fps 29.80 0.870 15.68
pretrained mine 29.77 0.874 15.58
SuperSloMo 30.22 0.880 15.18

Prerequisites

This codebase was developed and tested with pytorch 0.4.1 and CUDA 9.2.
Install:

For GPU, run

conda install pytorch=0.4.1 cuda92 -c pytorch
pip install torchvision

For CPU, run

conda install pytorch-cpu=0.4.1 -c pytorch
pip install torchvision

Training

Preparing training data

In order to train the model using the provided code, the data needs to be formatted in a certain manner.
The create_dataset.py script uses ffmpeg to extract frames from videos.

Adobe240fps

For adobe240fps, download the dataset, unzip it and then run the following command

python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset --dataset adobe240fps

Custom

For custom dataset, run the following command

python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset

The default train-test split is 90-10. You can change that using command line argument --train_test_split.

Run the following commmand for help / more info

python data\create_dataset.py --h

Training

In the train.ipynb, set the parameters (dataset path, checkpoint directory, etc.) and run all the cells.

or to train from terminal, run:

python train.py --dataset_root path\to\dataset --checkpoint_dir path\to\save\checkpoints

Run the following commmand for help / more options like continue from checkpoint, progress frequency etc.

python train.py --h

Tensorboard

To get visualization of the training, you can run tensorboard from the project directory using the command:

tensorboard --logdir log --port 6007

and then go to https://localhost:6007.

Evaluation

Pretrained model

You can download the pretrained model trained on adobe240fps dataset here.

Video Converter

You can convert any video to a slomo or high fps video (or both) using video_to_slomo.py. Use the command

python video_to_slomo.py --ffmpeg path\to\folder\containing\ffmpeg --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mp4

If you want to convert a video from 30fps to 90fps set fps to 90 and sf to 3 (to get 3x frames than the original video).

Run the following commmand for help / more info

python video_to_slomo.py --h

More info TBA

To-Do's:

Task Status
Add evaluation script for UCF dataset TBD
Add getting started guide TBD
Add video converter script Done
Add pretrained model Done