Skip to content
Official Pytorch implementation of "Learnable Gated Temporal Shift Module for Deep Video Inpainting. Chang et al. BMVC 2019."
Branch: master
Clone or download
Latest commit 853aeb3 Jul 8, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github First commit Jul 8, 2019
dataset First commit Jul 8, 2019
doc First commit Jul 8, 2019
libs First commit Jul 8, 2019
src First commit Jul 8, 2019
.flake8 First commit Jul 8, 2019
.gitignore First commit Jul 8, 2019
.gitmodules First commit Jul 8, 2019
.travis.yml First commit Jul 8, 2019
README.md First commit Jul 8, 2019
environment.yaml First commit Jul 8, 2019

README.md

Learnable Gated Temporal Shift Module for Deep Video Inpainting

Official Pytorch implementation of "Learnable Gated Temporal Shift Module for Deep Video Inpainting. Chang et al. BMVC 2019." arXiv

This repository also includes the implementation of some baselines and the Free-form Video Inpainting (FVI) dataset in "Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN. Chang et al. arXiv 2019." arXiv

teaser

See YouTube video demo or full resolution videos on Google Drive

Introduction

In "Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN. Chang et al. arXiv 2019.", we proposed 3D gated convolutions, Temporal PatchGAN and mask video generation algorithm to deal with free-form video inpainting in an end-to-end way. It is the first deep method for free-form video inpainting and achieves state-of-the-art performance both quantitatively and qualitatively. However, there are too many parameters for 3D gated convolutions and it takes long to train and inference.

Therefore, in "Learnable Gated Temporal Shift Module for Deep Video Inpainting. Chang et al. BMVC 2019.", we proposed a new LGTSM based on temporal shift module (TSM) for action recognition to reduce model parameters and training time to about 33%. The performance is almost the same as our previous work.

lgtsm

block

This repository contains source code for both works. Some pretrained weights for the GTSM one are given, while the LGTSM code could be found in the LGTSM branch. The pretrained weights of the 3DGated model is not available now due to some compatibility issues. The implementation of the baseline CombCN is also provided.

compare

Environment Setup

git clone git@github.com:amjltc295/Free-Form-Video-Inpainting.git
cd Free-Form-Video-Inpainting
git submodule update --init --recursive
conda env create -f environment.yaml
source activate free_form_video_inpainting

Training

Please see training

Testing

  1. Download corresponding pretrained weights from Google Drive
  2. Update parameters in src/other_configs/inference_example.json:
    • If you want to test on other data, set root_masks_dir for testing masks and root_videos_dir for testing frames.
    • If you want to turn on evaluation, set evaluate_score to true.
  3. Run
python train.py -r <pretrained_weight_path> --dataset_config other_configs/inference_example.json -od test_outputs

Then, you should have a directory src/test_outputs/ like:

test_outputs
└── epoch_0
    ├── test_object_like
    │   ├── inputs
    │   │   └── input_0000
    │   └── result_0000
    └── test_object_removal
        ├── inputs
        │   └── input_0000
        └── result_0000

The following GIFs show the figures that will appear in

(top row) test_object_like/result_0000, test_object_like/inputs/result_0000,

(bottom row) test_object_removal/result_0000, test_object_removal/inputs/result_0000

License

This repository is limited to research purpose. For any commercial usage, please contact us.

Authors

Ya-Liang Chang (Allen) amjltc295 yaliangchang@cmlab.csie.ntu.edu.tw

Zhe-Yu Liu Nash2325138 zhe2325138@cmlab.csie.ntu.edu.tw

Please cite our papers if you use this repo in your research:

@article{chang2019free,
  title={Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN},
  author={Chang, Ya-Liang and Liu, Zhe Yu and Lee, Kuan-Ying and Hsu, Winston},
  journal={arXiv preprint arXiv:1904.10247},
  year={2019}
}
@article{chang2019learnable,
  title={Learnable Gated Temporal Shift Module for Deep Video Inpainting"},
  author={Chang, Ya-Liang and Liu, Zhe Yu and Lee, Kuan-Ying and Hsu, Winston},
  journal={BMVC},
  year={2019}
}
You can’t perform that action at this time.