2021.3.9 News: We have updated our arXiv paper and fixed most of the defects.
2021.2.9 News: We have updated the RIFEv2 model, faster and much better! Please check our Update Log.
Our model can run 30+FPS for 2X 720p interpolation on a 2080Ti GPU. Currently, our method supports 2X,4X,8X... interpolation, and multi-frame interpolation between a pair of images. Everyone is welcome to use our alpha version and make suggestions!
16X interpolation results from two input images:
Software Squirrel-RIFE(中文软件) | Waifu2x-Extension-GUI | Flowframes | RIFE-ncnn-vulkan | RIFE-App(Paid) | Autodesk Flame |
2d Animation DAIN-App vs RIFE-App | Chika Dance | 御坂大哥想让我表白 - 魔女之旅 | ablyh - 超电磁炮 超电磁炮.b | 赫萝与罗伦斯的旅途 - 绫波丽 | 花儿不哭 - 乐正绫 |
没有鼠鼠的雏子Official - 千恋万花 | 晨曦光晖 - 从零开始的异世界生活 | 琴乃乃 - 天才麻将少女 |
3d Animation 没有鼠鼠的雏子Official - 原神 | 今天我练出腹肌了吗 - 最终幻想 仙剑奇侠传 | 娜不列颠 - 冰雪奇缘 |
MV and Film Navetek - 邓丽君 | 生米阿怪 - 周深 | EzioAuditoreDFirenZe - 中森明菜 | Life in a Day 2020 |
MMD 深邃黑暗の银鳕鱼 - 镜音铃 fufu fufu.b | Abism0 - 弱音 |
git clone git@github.com:hzwer/arXiv2020-RIFE.git
cd arXiv2020-RIFE
pip3 install -r requirements.txt
-
Download the pretrained HDv2 models from here. (百度网盘链接:https://pan.baidu.com/s/1EvQCA8Y7LuDe9KE6XICpXA 密码:xm3z，把压缩包解开后放在 train_log/*.pkl)
-
Unzip and move the pretrained parameters to train_log/*.pkl
Video Frame Interpolation
You can use our demo video or your own video.
python3 inference_video.py --exp=1 --video=video.mp4
(generate video_2X_xxfps.mp4)
python3 inference_video.py --exp=2 --video=video.mp4
(for 4X interpolation)
python3 inference_video.py --exp=1 --video=video.mp4 --scale=0.5
(If your video has very high resolution such as 4K, we recommend set --scale=0.5 (default 1.0). If you generate disordered pattern on your videos, try set --scale=2.0. This parameter control the process resolution for optical flow model.)
python3 inference_video.py --exp=2 --img=input/
(to read video from pngs, like input/0.png ... input/612.png, ensure that the png names are numbers)
python3 inference_video.py --exp=2 --video=video.mp4 --fps=60
(add slomo effect, the audio will be removed)
python3 inference_video.py --video=video.mp4 --montage --png
(if you want to montage the origin video, skip static frames and save the png format output)
The warning info, 'Warning: Your video has *** static frames, it may change the duration of the generated video.' means that your video has changed the frame rate by adding static frames, it is common if you have processed 25FPS video to 30FPS.
Image Interpolation
python3 inference_img.py --img img0.png img1.png --exp=4
(2^4=16X interpolation results) After that, you can use pngs to generate mp4:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0
You can also use pngs to generate gif:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf "split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1" output/slomo.gif
Place the pre-trained models in train_log/\*.pkl (as above)
Building the container:
docker build -t rife -f docker/Dockerfile .
Running the container:
docker run --rm -it -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4
docker run --rm -it -v $PWD:/host rife:latest inference_img --img img0.png img1.png --exp=4
Using gpu acceleration (requires proper gpu drivers for docker):
docker run --rm -it --gpus all -v /dev/dri:/dev/dri -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4
Download RIFE model reported by our paper.
UCF101: Download UCF101 dataset at ./UCF101/ucf101_interp_ours/
Vimeo90K: Download Vimeo90K dataset at ./vimeo_interp_test
MiddleBury: Download MiddleBury OTHER dataset at ./other-data and ./other-gt-interp
HD: Download HD dataset at ./HD_dataset
python3 benchmark/UCF101.py
# "PSNR: 35.246 SSIM: 0.9691"
python3 benchmark/Vimeo90K.py
# "PSNR: 35.506 SSIM: 0.9779"
python3 benchmark/MiddleBury_Other.py
# "IE: 1.962"
python3 benchmark/HD.py
# "PSNR: 31.99"
python3 benchmark/HD_multi.py
# "PSNR: 18.89(544*1280), 28.83(720p), 24.96(1080p)"
Because Vimeo90K dataset and the corresponding optical flow labels are too large, we cannot provide a complete dataset download link. We provide you with a subset containing 100 samples for testing the pipeline. Please unzip it at ./dataset
Each sample includes images (I0 I1 Imid : 9 x 256 x 448), and optical flow (flow_t0, flow_t1: 4, 256, 448).
For origin images, you can download them from Vimeo90K dataset.
For generating optical flow labels, our paper use pytorch-liteflownet. In our trainning, we use the optical flow labels generated on 2X size images using official released model. You can also generate labels during training, or finetune the optical flow network on your training set.
We use 16 CPUs, 4 GPUs and 20G memory for training:
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --world_size=4
@article{huang2020rife,
title={RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
journal={arXiv preprint arXiv:2011.06294},
year={2020}
}
Optical Flow: ARFlow pytorch-liteflownet RAFT pytorch-PWCNet
Video Interpolation: DVF TOflow SepConv DAIN CAIN MEMC-Net SoftSplat BMBC EDSC EQVI
感谢支持 Paypal Sponsor: https://www.paypal.com/paypalme/hzwer
