TAVA: Template-free Animatable Volumetric Actors

Ruilong Li · Julian Tanke · Minh Vo · Michael Zollhoefer
Jürgen Gall · Angjoo Kanazawa · Christoph Lassner

ECCV 2022

Given multiple sparse video views as well as 3D poses as inputs, TAVA creates a virtual actor consists of implicit shape, apperance, skinning weights in the canonical space, which is ready to be animated and rendered even with out-of-distribution poses. Dense correspondences across views and poses can also be established, which enables content editing during rendering. Without requiring body template, TAVA can be directly used for creatures beyond human as along as the skeleton can be defined.

---

Installation

conda create -n tava python=3.9
conda activate tava

conda install pytorch cudatoolkit=11.3 -c pytorch

python setup.py develop

Running the code

• See docs/dataset.md on how to accquire data used in this paper.
• See docs/experiment.md on using this code base for training and evaluation.
• See docs/benchmark.md for replication of our paper.

Toolbox

• tools/extract_mesh_weights.ipynb: Extracting mesh and skinning weights from the learnt canonical space.

• tools/eval_corr.ipynb: Evaluate the pixel-to-pixel correspondences using ground-truth mesh on Animal Data.

License

BSD 3-clause (see LICENSE.txt).

Misc

• Training NeRF on NeRF synthetic dataset: See docs/nerf.md for instructions.
• Training Mip-NeRF on NeRF synthetic dataset: See docs/mipnerf.md for instructions.

Citation

@inproceedings{li2022tava,
  title={TAVA: Template-free animatable volumetric actors},
  author={Li, Ruilong and Tanke, Julian and Vo, Minh and Zollhofer, Michael and Gall, Jurgen and Kanazawa, Angjoo and Lassner, Christoph},
  journal={European Conference on Computer Vision (ECCV)},
  year={2022}
}