for Guiding Class-Conditional Video Prediction
Unsupervised Keypoint Learning An official implementation of the paper "Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction", NeurIPS, 2019. [paper] [supp]
I. Requirements
- Linux
- NVIDIA GeForce GTX 1080Ti
- Tensorflow 1.12.0
- Python3 (>= 3.5.2)
Dependencies
You can install packages by running pip install -r requirements.txt.
Or you can download our prebuilt docker image,
by running docker pull join16/python3-cuda:3.5-cuda9.0-nips2019
.
If you want, you can build docker image manually,
by running docker build -t {image_name} .
※ Dataset
This code is for the Penn Action dataset. The dataset can be downloaded here. After download PennAction.tar.gz, unzip and then run following code to prepare dataset.
./prepare_penn_dataset.sh {unzipped_original_dataset_dir}
※ Pretrained VGG-Net
For the training, pretrained VGG19 network is needed. It can be downloaded here.
II. Train
※※※ Please adjust the paths for inputs and outputs in the configuration file. ※※※
1. Train the keypoints detector & image translator
python train.py --mode detector_translator --config configs/penn.yaml
2. Make pseudo-keypoints labels
python make_pseudo_labels.py --config configs/penn.yaml --checkpoint {path/to/detector_translator/checkpoint}
3. Train the motion generator
python train.py --mode motion_generator --config configs/penn.yaml
III. Test
python evaluate.py --config configs/penn.yaml \
--checkpoint_stage1 {path/to/detector_translator/checkpoint} \
--checkpoint_stage2 {path/to/motion_generator/checkpoint} \
--save_dir {path/to/save/results}
Pretrained model
IV. Results
※※※ All videos were generated from a single input image. ※※※
Penn Action
UvA-NEMO
MGIF
※※※ Qualitative comparison of the results. ※※※
V. Related Works
Learning to Generate Long-term Future via Hierarchical Prediction, Villegas et. al., ICML, 2017. [code]
Hierarchical Long-term Video Prediction without Supervision, Wichers et. al., ICML, 2018. [code]
Flow-Grounded Spatial-Temporal Video Prediction from Still Images, Li et. al., ECCV, 2018. [code]
※ Citation
Please cite our paper when you use this code.
@inproceedings{yunji_neurips_2019,
title={Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction},
author={Kim, Yunji and Nam, Seonghyeon and Cho, In and Kim, Seon Joo},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2019}
}