Skip to content

PyTorch implementation of "Improved Modeling of 3D Shapes with Multi-view Depth Maps". 3DV 2020

Notifications You must be signed in to change notification settings

kampta/multiview-shapes

Repository files navigation

Improved Modeling of 3D Shapes with Multi-view Depth Maps

PyTorch implementation of Improved Modeling of 3D Shapes with Multi-view Depth Maps. We propose a novel encoder-decoder generative model for 3D shapes using multi-view depth maps; and show applications in single-view reconstruction, shape generation and 3D shape interpolation. Accepted as an Oral paper in 3DV.

arXiv

Short Video

Long Video

Illustration

If you use this code in your work, please cite

@inproceedings{gupta2020improved,
  title={{Improved Modeling of 3D Shapes with Multi-view Depth Maps}},
  author={Gupta, Kamal and Jabbireddy, Susmija and Shah, Ketul and Shrivastava, Abhinav and Zwicker, Matthias},
  booktitle={International Conference on 3D Vision},
  year={2020},
  url={http://multiview-shapes.umiacs.io},
}

Data preparation

In all our experiments, we use ShapeNetCore (v2) dataset. You'll need to sign an agreement to access and download the dataset on the linked webpage.

Refer to dataset/README.md for details on data generation.

Training

Then model training is done in two steps. In the first step, we train an encoder-decoder architecture which takes a depthmap as input to encoder. The latent representation learned by encoder is used by view-conditioned generator which generates depthmap from any given viewpoint. Note that stylegan2 generator model is borrowed from this repo.

In the second step, we train an Implicit Maximum Likelihood Expectation (IMLE) model on top of the latent codes learned in the first step.

Training the encoder-decoder framework

python train_ae_avg.py /path/to/data \
    --ckpt_save_directory /path/to/output/model/checkpoints \
    --sample_save_directory /path/to/output/samples \
    --wandb --no_noise \
    --categories airplane \
    --d_reg_every 0 --g_reg_every 0 \
    --iter 75001 --batch 24 --num_vps 2 \
    --loader_type blender --load_sil --lr 0.004 \
    --channel_multiplier 1 --bins 256 --size 64 --val_batch 8 \
    --use_pretrained_if_available \
    --exp_name airplane_blender_base_avg_cm1_bins_256_size_64

Training the IMLE

Dump the latent codes of the model learned in the previous step and train IMLE as following

python train_imle.py

Evaluation

We have borrowed the code for computing CD and CMD from this repo. Please refer to the link on how to compile the code. For computing Chamfer Distance (CD):

python evaluate_recon_cd.py \
    --gt_pts_path /path/to/groundtruth/pointclouds \
    --gen_pts_path /path/to/reconstructed/pointclouds

For computing Earth Mover Distance (EMD):

python evaluate_recon_emd.py \
    --gt_pts_path /path/to/groundtruth/pointclouds \
    --gen_pts_path /path/to/reconstructed/pointclouds

License

MIT

About

PyTorch implementation of "Improved Modeling of 3D Shapes with Multi-view Depth Maps". 3DV 2020

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •