Skip to content
Tensorflow implementation of Learning-based Video Motion Magnification
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
configs Training Code Oct 12, 2018
.gitignore Running code only Jul 31, 2018
LICENSE Include Training Instruction Oct 12, 2018 Training Code Oct 12, 2018 Training Code Oct 12, 2018 Training Code Oct 12, 2018 Running code only Jul 31, 2018 Running code only Jul 31, 2018 Running code only Jul 31, 2018 Training Code Oct 12, 2018
requirements.txt Running code only Jul 31, 2018 Running code only Jul 31, 2018 Running code only Jul 31, 2018 Training Code Oct 12, 2018

Learning-based Video Motion Magnification

Tensorflow implementation of Learning-based Video Motion Magnification. [Project page] [Paper] [Data]

Collaborators: *(Tae-Hyun Oh, Ronnachai "Tiam" Jaroensri), Changil Kim, Mohamed A. Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik

*Equal contribution.

[Video demo]

Video demo


Install required packages:

sudo apt-get install python-dev 	# required to install setproctitle
pip install -r requirements.txt

This code has been tested with Tensorflow 1.3 and 1.8, CUDA 8.5 and 9.1, Ubuntu 14.04 and 16.04, although we expect it to work with any newer versions of Tensorflow and CUDA.


Download and place the pre-trained model in to data/. Only checkpoint is included in the package, and should be under data/training/o3f_hmhm2_bg_qnoise_mix4_nl_n_t_ds3/checkpoint.

Place video frames under data/vids/<video_name>, and run one of the two scripts:

# Two-frame based processing (Static, Dynamic mode)
sh <experiment_name> <video_name> <amplification_factor> <run_dynamic_mode>
# Temporal filtering based processing (frequency band pass filter)
sh <experiment_name> <video_name> <amplification_factor> <low_cutoff> <high_cutoff> <sampling_rate> <n_filter_tap> <filter_type>

Here, Dynamic mode stands for consequtive frame processing. Our code supports {"fir", "butter", "differenceOfIIR"} filters. For FIR filters, the filtering could be done both in Fourier domain and in temporal domain.

For example, for the baby video, you need to extract video frames into PNG files. You can run these commands from the repo directory (assuming the frames are under data/vids/baby):

# Static mode, using first frame as reference.
sh o3f_hmhm2_bg_qnoise_mix4_nl_n_t_ds3 baby 10
# Dynamic mode, magnify difference between consecutive frames.
sh o3f_hmhm2_bg_qnoise_mix4_nl_n_t_ds3 baby 10 yes
# Using temporal filter (same as Wadhawa et al.)
sh o3f_hmhm2_bg_qnoise_mix4_nl_n_t_ds3 baby 20 0.04 0.4 30 2 differenceOfIIR

For your reference, you can use this command to extract baby video frames (replace avconv with ffmpeg as necessary):

avconv -i <path_to_input>/baby.mp4 -f image2 <path_to_output>/baby/%06d.png

For custom video and output location, you should look into modifying the two scripts directly. For custom checkpoint location, look in the configuration file under configs/. There should be an option for checkpoint location that you can edit.


Use the utilities to convert your dataset into tfrecords, or download the pre-processed tfrecords (see here).

Set the data path in the config file, and run the training script:

bash configs/<your config file>.conf

Be sure to configure your GPU appropriately (e.g. export CUDA_VISIBLE_DEVICES=...).


If you use this code, please cite our paper:

  title={Learning-based Video Motion Magnification},
  author={Oh, Tae-Hyun and Jaroensri, Ronnachai and Kim, Changil and Elgharib, Mohamed and Durand, Fr{\'e}do and Freeman, William T and Matusik, Wojciech},
  journal={arXiv preprint arXiv:1804.02684},

This implementation is provided for academic use only.


The authors would like to thank Toyota Research Institute, Shell Research, and Qatar Computing Research Institute for their generous support of this project. Changil Kim was supported by a Swiss National Science Foundation fellowship P2EZP2 168785.

Utilities and skeleton code borrowed from tensorflow implementation of CycleGAN

You can’t perform that action at this time.