Skip to content

xuehy/HiFiFace-pytorch

Repository files navigation

An unofficial implementation of HiFiFace

This is an unofficial implementation of HiFiFace, including some modifications.

Reference

Wang, Yuhan, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. “HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping.” arXiv, June 18, 2021. https://doi.org/10.48550/arXiv.2106.09965.

Results

Standard model from the original paper

results/exp_230901_base_1693564635742_320000_1.jpg results/p1.png results/p2.png results/p3.png results/p4.png results/p5.png

Model with eye and mouth heat map loss

to better preserve gaze and mouth shape /results/origan-v0-new-3d-250k-eye-mouth-hm-weight-10k-10k_1685515837755_190000_1.jpg

Dependencies

Models

The project depends on multiple exterior repositories whose codes have been integrated into this repo already. You need to download corresponding model files.

In configs/train_config.py, modify the dict to the correct paths of your downloaded models:

identity_extractor_config = {
        "f_3d_checkpoint_path": "/data/useful_ckpt/Deep3DFaceRecon/epoch_20_new.pth",
        "f_id_checkpoint_path": "/data/useful_ckpt/arcface/ms1mv3_arcface_r100_fp16_backbone.pth",
        "bfm_folder": "/data/useful_ckpt/BFM",
        "hrnet_path": "/data/useful_ckpt/face_98lmks/HR18-WFLW.pth",
    }

Download urls:

Environments

  • PyTorch >= 2.0
  • torchaudio with cuda support
  • ffmpeg
  • gradio
  • etc.

You can also use the pre-built docker image: docker pull xuehy93/hififace

Datasets

We use a mixed dataset from VGGFace2 HQ, Asia-celeb and some other data collected by ourselves. If you want to train your own model, you may download VGGFace2 HQ and Asia-celeb datasets.

Pretrained models

We provide two pretrained models here , one is trained without gaze and mouth shape loss and the other is trained with attributes loss:

  1. standard_model
  2. with_gaze_and_mouth

Usage

Inference

a demo based on gradio is provided: app/app.py

You need to download all auxiliary models and pretrained models, then modify the model paths in app/app.py and configs/train_config.py

Train

see entry/train.py

HuggingFace Inference Demo

We provide inference demo for image based face swap on huggingface with both models: HiFiFace-inference-demo

Modifications

We find that the attributes of the faces such as eye gaze directions and mouth shapes cannot be well preserved after face swapping, and therefore we introduce some auxiliary losses including the eye heat map loss and the mouth heatmap loss.

You can enable them or disable them by setting eye_hm_loss and mouth_hm_loss in configs/train_config.py

Acknowledgements

  1. The official repo https://github.com/johannwyh/HifiFace although they provide no codes, the disccusions in the issues are helpful.

Problems

Currently our implementation is not perfect:

  1. The attributes such as gaze and mouth shapes cannot be well preserved with the original model proposed in the paper. This problem is obvious in video face swapping results.
  2. With the eye and mouth heat map losses, the attributes are better preserved, however, the face swap similarity drops significantly.

Any dicussions and feedbacks are welcome!!!

About

An unofficial implementation of HiFiFace

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published