This repository provides code for "Neural Texture Puppeteer: A Framework for Neural Geometry and Texture Rendering of Articulated Shapes, Enabling Re-Identification at Interactive Speed" (WACV Workshop "CV4Smalls" 2024, oral).
Abstract
In this paper, we present a neural rendering pipeline for textured articulated shapes that we call Neural Texture Puppeteer. Our method separates geometry and texture encoding. The geometry pipeline learns to capture spatial relationships on the surface of the articulated shape from ground truth data that provides this geometric information. A texture auto-encoder makes use of this information to encode textured images into a global latent code. This global texture embedding can be efficiently trained separately from the geometry, and used in a downstream task to identify individuals. The neural texture rendering and the identification of individuals run at interactive speeds. To the best of our knowledge, we are the first to offer a promising alternative to CNN- or transformer-based approaches for re-identification of articulated individuals based on neural rendering. Realistic looking novel view and pose synthesis for different synthetic cow textures further demonstrate the quality of our method. Restricted by the availability of ground truth data for the articulated shape’s geometry, the quality for real-world data synthesis is reduced. We further demonstrate the flexibility of our model for real-world data by applying a synthetic to real-world texture domain shift where we reconstruct the texture from a real-world 2D RGB image. Thus, our method can be applied to endangered species where data is limited. Our novel synthetic texture dataset NePuMoo is publicly available to inspire further development in the field of neural rendering-based re-identification.
If you find a bug, have a question or know how to improve the code, please open an issue.
Set up a conda environment with conda env create -f environment.yml.
Our novel NePuMoo data set can be downloaded here. Unzip and copy the "multiview_cow" folder to ./data/.
Pre-trained weights for cows (our novel NePuMoo data set), the Human3.6M dataset, and our ablation studies can be downloaded here. Unzip and copy the "experiments" folder to ./.
The pipeline for geometry can render depth maps, masks, and NNOPCS maps.
Novel pose synthesis
To render multiple views of the NePuMoo test set, run:
python test.py -exp_name geometry_cows -checkpoint 1950
where -exp_name specifies the name of the experiment, and -checkpoint the epoch of the trained weights.
Novel pose and view synthesis
To render multiple novel views of the NePuMoo test set, run:
python test_nv.py -exp_name geometry_cows -checkpoint 1950 -novel_views_cfg configs/parameters_ref.cfg
where -novel_views_cfg specifies the path to the configuration file containing the novel views, and the other command line arguments are the same as above.
The texture pipeline can render color maps.
Novel pose synthesis
To render multiple views of the NePuMoo test set, run:
python test_texture.py -exp_name texture_cows -checkpoint 535
where the command line arguments are the same as above.
Novel pose and view synthesis
To render multiple novel views of the NePuMoo test set, run:
python test_texture_nv.py -exp_name texture_cows -checkpoint 535 -novel_views_cfg configs/parameters_ref.cfg
where the command line arguments are the same as above.
Preliminary tasks
Clone this repository into ./data/h36m/.
Download and unzip the 3D poses (positions, not angles) of the action "Posing" for all subjects (you need a login to download). If you picked correctly, the folder you downloaded is named "Poses_D3_Positions_Posing". Copy that folder to .data/h36m/data/.
We use the processed data from AniNeRF in ./data/h36m/data/.
Novel pose synthesis
To render multiple views of the Human3.6M test set, run:
python test_texture.py -exp_name texture_h36m -checkpoint 2080
where the command line arguments are the same as above.
To get the global texture embedding for the known views, run:
python get_textures.py -exp_name_texture texture_cows -checkpoint_texture 535 -data cows -exp_name_geometry geometry_cows -checkpoint_geometry 1950
where the command line arguments are similar as above.
To get the global texture embedding for novel views and occlusions, run get_textures_nv.py and get_textures_occ.py respectively instead.
To train the geometry pipeline, run:
python train.py -exp_name EXP_NAME -cfg_file CFG_FILE -data DATA_TYPE
where EXP_NAME is the name of the experiment you choose, CFG_FILE is the path to a YAML configuration file, like these here, and DATA_TYPE can be cows or h36m.
To train the texture pipeline, run:
python train_texture.py -exp_name EXP_NAME -cfg_file CFG_FILE
where the command line arguments are the same as above. You need trained weights of the geometry pipeline for the same articulated shape.
@inproceedings{waldmann2024netepu,
title = {Neural Texture Puppeteer: A Framework for Neural Geometry and Texture Rendering of Articulated Shapes, Enabling Re-Identification at Interactive Speed},
author = {Waldmann, Urs and Johannsen, Ole and Goldluecke, Bastian},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops},
month = {January},
year = {2024},
pages = {69-79}
}