[NDF: Neural Deformable Fields for Dynamic Human Modelling]
Ruiqi Zhang, Jie Chen
ECCV 2022
This is the official implementation of NDF published at ECCV2022. The proposed method is able to conduct novel view synthesis and novel pose synthesis given the multi-view video observation. The above figure shows the dynamics related to changing poses. The code is implemented in pytorch and run on CentOS 7.7.
The learned rendering in observation space:
The learned rendering in NDF space:
conda create -n ndf_release python=3.7
conda activate ndf_release
pip install -r requirement.txt
# install blender for python
wget https://github.com/TylerGubala/blenderpy/releases/download/v2.91a0/bpy-2.91a0-cp37-cp37m-manylinux2014_x86_64.whl
pip install bpy-2.91a0-cp37-cp37m-manylinux2014_x86_64.whl
bpy_post_install
# install pyembree to accelerate trimesh
conda install -c conda-forge pyembree
Download SMPL UV from here and put it at data/smpl_uv_20200910/smpl_uv.obj.
Download ZJU-Mocap dataset from here and put it to data/zju_mocap/CoreView_313.
We use subject 313 as a running example.
We need to find the UV coordinate for each projected point. This can be done on-the-fly but to save time we pre-process all the transformation matrixs from SMPL surface in 3D to UV coordinate in UV space.
python tools/createDataset.py
Download latest.pth from here and put it at data/trained_model/if_nerf/ndf_313/latest.pth.
Then run
sh run_train_test_fid.sh 60 False
The evaluation result is expected to be
# novel view synthesis
mse: 0.0011499698173650414
psnr: 29.927725532762025
ssim: 0.9683078237882652
# novel pose synthesis
mse: 0.005014139546091728
psnr: 23.463418908827254
ssim: 0.9039105207522721
We train the model on a single V100 GPU. It consumes about 20 hours to finish the training.
export exp_name=ndf_313
sh run_train_test_fid.sh 60 False
The implementation took reference from Neural Body. We thank the authors for their generosity to release code and data.