LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control (Modified Version)

This project is a fork of LivePortrait.

NOTE - It's too late to merge with the original repo since there are so many changes to keep track of them.

Added Features

In this forked version, we have implemented the following major enhancements:

1. Server/Client Model Implementation:

   • We have separated the feature extraction and video generation components, making them transmittable over a network.
   • This enables distributed processing and remote execution capabilities.

2. Feature Reusability:

   • We've implemented the ability to save extracted features as separate files.
   • These saved feature files can be reused, significantly improving efficiency for repetitive processing tasks.

Differences from the Original Project

• Network-based Processing: Unlike the original project, this version allows for feature extraction and video generation processes to be handled separately over a network.
• Feature Storage and Reuse: This version introduces the ability to save and reuse intermediate results (features), which was not possible in the original implementation.

Usage

(Add detailed instructions here on how to use the new features. For example: how to run the server, how to use the client, how to save and load feature files, etc.)

Server/Client

1. Server

python server.py -s assets\examples\source\s2.jpg

2. Client from video file

python client.py -d assets\examples\driving\d0.mp4 -l 2

3. Client from Cam

python client_cam.py

Feature Save

1. Feature extraction

python separated.py -s assets\examples\source\s0.jpg -d assets\examples\driving\d0.mp4

2. Video generation using saved feature

python separated.py -s assets\examples\source\s0.jpg -d output\extracted_features.pkl

---

The original README content follows:

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Jianzhu Guo ^1†  Dingyun Zhang ^1,2  Xiaoqiang Liu ¹  Zhizhou Zhong ^1,3  Yuan Zhang ¹ 

Pengfei Wan ¹  Di Zhang ¹ 

¹ Kuaishou Technology  ² University of Science and Technology of China  ³ Fudan University 

🔥 For more results, visit our homepage 🔥

🔥 Updates

• 2024/07/04: 🔥 We released the initial version of the inference code and models. Continuous updates, stay tuned!
• 2024/07/04: 😊 We released the homepage and technical report on arXiv.

Introduction

This repo, named LivePortrait, contains the official PyTorch implementation of our paper LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.

🔥 Getting Started

1. Clone the code and prepare the environment

git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait

# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt

2. Download pretrained weights

Download our pretrained LivePortrait weights and face detection models of InsightFace from Google Drive or Baidu Yun. We have packed all weights in one directory 😊. Unzip and place them in ./pretrained_weights ensuring the directory structure is as follows:

pretrained_weights
├── insightface
│   └── models
│       └── buffalo_l
│           ├── 2d106det.onnx
│           └── det_10g.onnx
└── liveportrait
    ├── base_models
    │   ├── appearance_feature_extractor.pth
    │   ├── motion_extractor.pth
    │   ├── spade_generator.pth
    │   └── warping_module.pth
    ├── landmark.onnx
    └── retargeting_models
        └── stitching_retargeting_module.pth

3. Inference 🚀

python inference.py

If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image, and generated result.

Or, you can change the input by specifying the -s and -d arguments:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4

# or disable pasting back
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# more options to see
python inference.py -h

More interesting results can be found in our Homepage 😊

4. Gradio interface

We also provide a Gradio interface for a better experience, just run by:

python app.py

5. Inference speed evaluation 🚀🚀🚀

We have also provided a script to evaluate the inference speed of each module:

python speed.py

Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with torch.compile:

Model	Parameters(M)	Model Size(MB)	Inference(ms)
Appearance Feature Extractor	0.84	3.3	0.82
Motion Extractor	28.12	108	0.84
Spade Generator	55.37	212	7.59
Warping Module	45.53	174	5.21
Stitching and Retargeting Modules	0.23	2.3	0.31

Note: the listed values of Stitching and Retargeting Modules represent the combined parameter counts and the total sequential inference time of three MLP networks.

Acknowledgements

We would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace repositories, for their open research and contributions.

Citation 💖

If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{guo2024live,
  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
  author  = {Jianzhu Guo and Dingyun Zhang and Xiaoqiang Liu and Zhizhou Zhong and Yuan Zhang and Pengfei Wan and Di Zhang},
  year    = {2024},
  journal = {arXiv preprint:2407.03168},
}