Tri2plane

The repository for 'Tri²-plane: Volumetric Avatar Reconstruction with Feature Pyramid'

[Tri²-plane: Volumetric Avatar Reconstruction with Feature Pyramid],
Arxiv Link: https://arxiv.org/abs/2401.09386,
Luchuan Song, Pinxin Liu, Lele Chen, Celong Liu and Chenliang Xu,
Arxiv 2024

Abstract

Recent years have witnessed considerable achievements in facial avatar reconstruction with neural volume rendering. Despite notable advancements, the reconstruction of complex and dynamic head movements from monocular videos still suffers from capturing and restoring fine-grained details. In this work, we propose a novel approach, named Tri²-plane, for monocular photo-realistic volumetric head avatar reconstructions. Distinct from the existing works that rely on a single tri-plane deformation field for dynamic facial modeling, the proposed Tri²-plane leverages the principle of feature pyramids and three top-to-down lateral connections tri-planes for details improvement. It samples and renders facial details at multiple scales, transitioning from the entire face to specific local regions and then to even more refined sub-regions. Moreover, we incorporate a camera-based geometry-aware sliding window method as an augmentation in training, which improves the robustness beyond the canonical space, with a particular improvement in cross-identity generation capabilities. Experimental outcomes indicate that the Tri²-plane not only surpasses existing methodologies but also achieves superior performance across both quantitative metrics and qualitative assessments through experiments.

Code (Coming Soon)

Video (Coming Soon)

Citation

@article{song2024tri,
  title={Tri $\^{}$\{$2$\}$ $-plane: Volumetric Avatar Reconstruction with Feature Pyramid},
  author={Song, Luchuan and Liu, Pinxin and Chen, Lele and Liu, Celong and Xu, Chenliang},
  journal={arXiv preprint arXiv:2401.09386},
  year={2024}
}