Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss （ATVGnet）

By Lele Chen , Ross K Maddox, Zhiyao Duan, Chenliang Xu.

University of Rochester.

Table of Contents

0. Introduction
1. Citation
2. Running
3. Model
4. Results
5. Disclaimer and known issues

Introduction

This repository contains the original models (AT-net, VG-net) described in the paper Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss. The demo video is avaliable at https://youtu.be/eH7h_bDRX2Q. This code can be applied directly in LRW and GRID. The outputs from the model are visualized here: the first one is the synthesized landmark from ATnet, the rest of them are attention, motion map and final results from VGnet.

Citation

If you use any codes, models or the ideas from this repo in your research, please cite:

@inproceedings{chen2019hierarchical,
  title={Hierarchical cross-modal talking face generation with dynamic pixel-wise loss},
  author={Chen, Lele and Maddox, Ross K and Duan, Zhiyao and Xu, Chenliang},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={7832--7841},
  year={2019}
}

Running

0. This code is tested under Python 2.7. The model we provided is trained on LRW. However, it works fine on GRID,VOXCELB and other datasets. You can directly compare this model on other dataset with your own model. We treat this as fair comparison.

1. Pytorch environment:Pytorch 0.4.1. (conda install pytorch=0.4.1 torchvision cuda90 -c pytorch)

2. Install requirements.txt (pip install -r requirement.txt)

3. Download the pretrained ATnet and VGnet weights at google drive. Put the weights under model folder.

4. Run the demo code: python demo.py

   • -device_ids: gpu id
   • -cuda: using cuda or not
   • -vg_model: pretrained VGnet weight
   • -at_model: pretrained ATnet weight
   • -lstm: use lstm or not
   • -p: input example image
   • -i: input audio file
   • -lstm: use lstm or not
   • -sample_dir: folder to save the outputs
   • ...

5. Download and unzip the training data from LRW

6. Preprocess the data (Extract landmark and crop the image by dlib).

7. Train the ATnet model: python atnet.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_dir: folder to save weights
   • -lstm: use lstm or not
   • -sample_dir: folder to save visualized images during training
   • ...

8. Test the model: python atnet_test.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_name: pretrained weights
   • -sample_dir: folder to save the outputs
   • -lstm: use lstm or not
   • ...

9. Train the VGnet: python vgnet.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_dir: folder to save weights
   • -sample_dir: folder to save visualized images during training
   • ...

10. Test the VGnet: python vgnet_test.py

    • -device_ids: gpu id
    • -batch_size: batch size
    • -model_name: pretrained weights
    • -sample_dir: folder to save the outputs
    • ...

Model

0. Overall ATVGnet

1. Regresssion based discriminator network

   

Results

0. Result visualization on different datasets:

   

1. Reuslt compared with other SOTA methods:

   

2. The studies on image robustness respective with landmark accuracy:

   

3. Quantitative results:

   

Disclaimer and known issues

0. These codes are implmented in Pytorch.
1. In this paper, we train LRW and GRID seperately.
2. The model are sensitive to input images. Please use the correct preprocessing code.
3. I didn't finish the data processing code yet. I will release it soon. But you can try the model and replace with your own image.
4. If you want to train these models using this version of pytorch without modifications, please notice that:
   • You need at lest 12 GB GPU memory.
   • There might be some other untested issues.
5. There is another intresting and useful research on audio to landmark genration. Please check it out at https://github.com/eeskimez/Talking-Face-Landmarks-from-Speech.

Todos

• Release training data

License

MIT