This is a Windows installation tutorial fork. The main code will not be updated and all credits belong to the original authors.

📖 Depth-Aware Generative Adversarial Network for Talking Head Video Generation (CVPR 2022)

🔥 If DaGAN is helpful in your photos/projects, please help to ⭐ it or recommend it to your friends. Thanks🔥

[Paper]   [Project Page]   [Demo]   [Poster Video]

Fa-Ting Hong, Longhao Zhang, Li Shen, Dan Xu
The Hong Kong University of Science and Technology

Cartoon Sample

cartoon.mp4

Human Sample

celeb.mp4

Voxceleb1 Dataset

🚩 Updates

• 🔥🔥✅ July 26, 2022: The normal dataparallel training scripts were released since some researchers informed me they ran into DistributedDataParallel problems. Please try to train your own model using this command. Also, we deleted the command line "with torch.autograd.set_detect_anomaly(True)" to boost the training speed.

• 🔥🔥✅ June 26, 2022: The repo of our face depth network is released, please refer to Face-Depth-Network and feel free to email me if you meet any problem.

• 🔥🔥✅ June 21, 2022: [Digression] I am looking for research intern/research assistant opportunities in European next year. Please contact me If you think I'm qualified for your position.

• 🔥🔥✅ May 19, 2022: The depth face model (50 layers) trained on Voxceleb2 is released! (The corresponding checkpoint of DaGAN will release soon). Click the LINK

• 🔥🔥✅ April 25, 2022: Integrated into Huggingface Spaces 🤗 using Gradio. Try out the web demo: (GPU version will come soon!)

• 🔥🔥✅ Add SPADE model, which produces more natural results.

🔧 Dependencies and Installation

• Use Anaconda
• NVIDIA GPU + CUDA

Installation

We now provide a clean version of DaGAN, which does not require customized CUDA extensions.

1. Clone repo

   git clone https://github.com/harlanhong/CVPR2022-DaGAN.git
   cd CVPR2022-DaGAN

2. Install dependent packages

   conda create -n DaGAN python=3.7 && conda activate DaGAN
   conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
   conda install pytorch torchvision torchaudio cpuonly -c pytorch  # for CPU only
   pip install -r requirements.txt
   cd face-alignment && python setup.py install && cd ..

⚡ Quick Inference

We take the paper version for an example. More models can be found here.

YAML configs

See config/vox-adv-256.yaml to get description of each parameter.

Pre-trained checkpoint

The pre-trained checkpoint of face depth network and our DaGAN checkpoints can be found under following link: OneDrive.

Inference! To run a demo, download checkpoint and run the following command: (add a --cpu flag if you are not running on CUDA/NVIDIA GPU)

For the normal checkpoint

python demo.py  --config config/vox-adv-256.yaml --driving_video driving/<video_file> --source_image input/<image_file> --checkpoint checkpoints/DaGAN_vox_adv_256.pth.tar --relative --adapt_scale --kp_num 15 --generator DepthAwareGenerator

For the SPADE checkpoint

python demo.py  --config config/vox-adv-256.yaml --driving_video driving/<video_file> --source_image input/<image_file> --checkpoint checkpoints/SPADE_DaGAN_vox_adv_256.pth.tar --relative --adapt_scale --kp_num 15 --generator SPADEDepthAwareGenerator

The result will be stored in result.mp4. The driving videos and source images should be cropped before it can be used in our method. To obtain some semi-automatic crop suggestions you can use python crop-video.py --inp some_youtube_video.mp4. It will generate commands for crops using ffmpeg.

This fork edits end here

💻 Training

Datasets

1. VoxCeleb. Please follow the instruction from https://github.com/AliaksandrSiarohin/video-preprocessing.

Train on VoxCeleb

To train a model on specific dataset run:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --master_addr="0.0.0.0" --master_port=12348 run.py --config config/vox-adv-256.yaml --name DaGAN --rgbd --batchsize 12 --kp_num 15 --generator DepthAwareGenerator

Or

CUDA_VISIBLE_DEVICES=0,1,2,3 python run_dataparallel.py --config config/vox-adv-256.yaml --device_ids 0,1,2,3 --name DaGAN_voxceleb2_depth --rgbd --batchsize 48 --kp_num 15 --generator DepthAwareGenerator

The code will create a folder in the log directory (each run will create a new name-specific directory). Checkpoints will be saved to this folder. To check the loss values during training see log.txt. By default the batch size is tunned to run on 8 GeForce RTX 3090 gpu (You can obtain the best performance after about 150 epochs). You can change the batch size in the train_params in .yaml file.

Also, you can watch the training loss by running the following command:

tensorboard --logdir log/DaGAN/log

When you kill your process for some reasons in the middle of training, a zombie process may occur, you can kill it using our provided tool:

python kill_port.py PORT

Training on your own dataset

1. Resize all the videos to the same size e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images. We recommend the later, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i/o performance.

2. Create a folder data/dataset_name with 2 subfolders train and test, put training videos in the train and testing in the test.

3. Create a config config/dataset_name.yaml, in dataset_params specify the root dir the root_dir: data/dataset_name. Also adjust the number of epoch in train_params.

📜 Acknowledgement

Our DaGAN implementation is inspired by FOMM. We appreciate the authors of FOMM for making their codes available to public.

📜 BibTeX

@inproceedings{hong2022depth,
            title={Depth-Aware Generative Adversarial Network for Talking Head Video Generation},
            author={Hong, Fa-Ting and Zhang, Longhao and Shen, Li and Xu, Dan},
            journal={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
            year={2022}
          }

📧 Contact

If you have any question or cooperation need (research purpose or commercial purpose), please email fhongac@cse.ust.hk.