Beyond 3DMM Space: Towards Fine-grained 3D Face Reconstruction
- The code and model of FGNet is released.
- The dataset of FG3D is released.
This repo releases the code and data in the paper published in ECCV 2020: Beyond 3DMM Space: Towards Fine-grained 3D Face Reconstruction (http://www.cbsr.ia.ac.cn/users/xiangyuzhu/papers/2020beyond.pdf).
The reconstruction results of widely used 3DMM lose the fine-grained geometry and look different from real faces. We propose a Fine-Grained reconstruction Network (FGNet) that can concentrate on shape modification and generate reconstruction results with fine-grained geometry.
- PyTorch >= 0.4.1 (PyTorch v1.1.0 is tested successfully on macOS and Linux.)
- Python >= 3.6 (Numpy, Scipy, Matplotlib)
- Dlib (Dlib is optionally for face and landmarks detection. There is no need to use Dlib if you can provide face bouding bbox and landmarks. Besides, you can try the two-step inference strategy without initialized landmarks.)
- OpenCV (Python version, for image IO operations.)
- Cython (For accelerating depth and PNCC render.)
- Platform: Linux or macOS (Windows is not tested.)
# installation structions sudo pip3 install torch torchvision # for cpu version. more option to see https://pytorch.org sudo pip3 install numpy scipy matplotlib sudo pip3 install dlib==19.5.0 # 19.15+ version may cause conflict with pytorch in Linux, this may take several minutes. If 19.5 version raises errors, you may try 19.15+ version. sudo pip3 install opencv-python sudo pip3 install cython
In addition, I strongly recommend using Python3.6+ instead of older version for its better design.
Clone this repo (this may take some time as it is a little big)
Download the following large models to models/
Build cython module (just one line for building)
cd MM3D/cython python3 setup.py build_ext -i
This is for accelerating depth estimation and PNCC render since Python is too slow in for loop.
main.pywith arbitrary image as input
python3 main.py -f examples/images/test.jpg
If you can see these output log in terminal, you run it successfully.
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning. Lossy conversion from float64 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning. 1/1 There are 1 images... => loading checkpoint 'models/_checkpoint_epoch_22.pth.tar' begin testing end
Fine-grained 3D Face (FG3D) Data
FG3D are constructed from FRGC, BP4D and CASIA-3D. We register and out-of-plane augment the three datasets, generating a large 3D dataset with 213,367 samples. Among FG3D, 90% subjects are used as the training set and the rest 10% subjects are the testing set.