AG3D: Learning to Generate 3D Avatars from 2D Image Collections (ICCV 2023)

Paper | Supplementary | Video | Project Page

Official code will be released soon for ICCV 2023 paper AG3D: Learning to Generate 3D Avatars from 2D Image Collections. Learned from 2D image collections, AG3D synthesizes novel 3D humans with high-quality appearance and geometry, different identities and clothing styles including loose clothing such as skirts.

News:

• [2023/09/13] Inference codes and evaluation are released!
• [2023/07/14] AG3D is accepted to ICCV 2023!

Who needs AG3D?

• If you want to train a 3D generative model of humans from your own 2D image collections.
• Our model can generate:
  • image
    • posed/canonical human RGB
    • posed/canonical human normal
    • raw rendering
  • mesh:
    • 3D corresponding clothed human geometry
  • animation
  • interpolation
  • novel view synthesis
• We also provide our pretrained models from DeepFashion and UBCFashion. You can easily finetune them with your own dataset.

Quick Start

Requirements

• 1–8 high-end NVIDIA GPUs. We have done all testing and development using V100 and A100 GPUs.
• 64-bit Python 3.8 and PyTorch 1.11.0 (or later). See https://pytorch.org for PyTorch install instructions.
• CUDA toolkit 11.3 or later.
• pytorch3d. See https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md for install instructions.

Clone this repo:

git clone https://github.com/zj-dong/AG3D.git
cd AG3D

We suggest to use anaconda to manage the python environments:

conda env create -f env.yml
conda activate ag3d
python setup.py install

Download Pose Distribution and Models

Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding locations:

mkdir training/deformers/smplx/SMPLX
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl training/deformers/smplx/SMPLX/SMPL_NEUTRAL.pkl

Download our pretrained models:

sh ./scripts/download_model.sh

Download pose distribution of training data:

sh ./scripts/download_pose.sh

Generate random samples

DeepFashion:

python test.py --network=./model/deep_fashion.pkl  --pose_dist=./data/dp_pose_dist.npy --output_path './result/deepfashion' --res=512 --truncation=0.7 --number=100 --type=gen_samples

UBCFashion:

python test.py --network=./model/ubc_fashion.pkl  --pose_dist=./data/ubc_pose_dist.npy --output_path './result/ubc' --res=512 --truncation=0.7 --number=100 --type=gen_samples

Inference

Generate results of novel view synthesis:

python test.py --network=./model/deep_fashion.pkl  --pose_dist=./data/dp_pose_dist.npy  --output_path='./result/gen_novel_view' --res=512 --truncation=0.7 --number=100 --type=gen_novel_view

Some sampled generated results are shown below.

Generate interpolation results:

python test.py --network=./model/deep_fashion.pkl   --pose_dist=./data/dp_pose_dist.npy  --output_path='./result/result_interp' --res=512 --truncation=0.7 --number=100 --type=gen_interp --is_mesh=True

Some sampled interpolation results:

Generate animations (please download motion sequence from AMASS):

python test.py --network=./model/deep_fashion.pkl   --pose_dist=./data/dp_pose_dist.npy  --output_path='./result/result_anim' --res=512 --truncation=0.7 --number=100 --type=gen_anim  --motion_path=./data/animation/motion_seq

Some sampled animation results:

Evaluation

Calculate metrics in the paper:

python evaluate.py --metrics=fid5k_full --network=./model/deep_fashion.pkl --data=./data/eva3d_icon.zip  --res=512 

Citation

If you find our code or paper useful, please cite as

@inproceedings{dong2023ag3d,
      title={{AG3D}: {L}earning to {G}enerate {3D} {A}vatars from {2D} {I}mage {C}ollections},
      author={Dong, Zijian and Chen, Xu and Yang, Jinlong and  Black, Michael J and Hilliges, Otmar and Geiger, Andreas},    
      booktitle   = {International Conference on Computer Vision (ICCV)},
      year      = {2023}
      }

Acknowledgements

We thank for their feedback and discussions.

Here are some great resources we benefit from:

• EG3D for the general 3D GAN structure and 3D representation
• PIFuHD, ICON for Normal Estimation and Preprocessing
• fast-SNARF for Human Deformation
• NeRF, StyleSDF, VolSDF for Volumetric Rendering
• smplx, PyMAF for Human Pose and Shape Estimation
• PaddleSeg and RVM for Background Removal
• DeepFashion and UBC Fashion for Dataset

This project was supported by the ERC Starting Grant LEGO-3D (850533), the BMWi project KI Delta Learning (project number 19A19013O) and the DFG EXC number 2064/1 - project number 390727645.