Parameterize Structure with Differentiable Template for 3D Shape Generation

This repository contains PyTorch implementation for "Parameterize Structure with Differentiable Template for 3D Shape Generation" (TVCG 2025).

Install

We recommend using conda to create a virtual environment with Python 3.8, PyTorch 1.10, CUDA 11.1.

conda create -n psdt python=3.8
conda activate psdt

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html 
pip install tqdm open3d pyvista shapely rich alpha_shapes ninja opencv-python tensorboard

To build PointNet2 lib, you need to:

cd util/pointnet2_ops_lib
python setup.py install

Data

Donwload the data.zip from Baidu Yun(code: 2fe8) or NJU Box(code: psdt). Unzip the file and put the data directory in the root directory of this repository. The data directory should be in the same directory as README.md. The data directory should include the following subdirectories: data, shapenet_obj and vis_list.

Weights

Download the weights.zip, weights.z01, weights.z02, weights.z03 from Baidu Yun(code: 2fe8) or NJU Box(code: psdt). Unzip the file and put the weights directory in the root directory of this repository. The weights directory should be in the same directory as README.md. The weights directory should include the following subdirectories: classify, gen_detail, gen_para and pnet.

Usage

Predict Structure

Train a model to predict the structure parameters of a 3D shape. (Section III.C.1)

Enter the pnet directory.

1. Train the model:

   python train.py --structure_name chair_4
   

2. Test the model:

   python test.py --structure_name chair_4 [--save_res]
   

   We provide the pretrained weights. You can directly run the test after downloading the weights.

Optimize Structure

The structure can be directly optimized. (Section III.C.1)

Enter the para_fit directory.

1. Optimize the structure:

   python fit.py fit --cate_name chair_4 --id 2b5953c986dd08f2f91663a74ccd2338         # run one case
   python fit.py fit --cate_name chair_4                                               # run all cases in the category
   

Reconstruct 3D Shape

Reconstruct a 3D shape given a point cloud. (Section III.C.1)

Enter the psdt_fit directory.

1. Reconstruct a 3D shape:

   python fit.py fit --cate_name chair_4 --id 2b5953c986dd08f2f91663a74ccd2338 --is_gen_mesh
   

   or you can run in two steps:

   python fit.py fit --cate_name chair_4 --id 2b5953c986dd08f2f91663a74ccd2338             # predict reconstruction information
   python fit.py gen_mesh --cate_name chair_4 --id 2b5953c986dd08f2f91663a74ccd2338        # generate mesh from reconstruction information
   

   The second step will employ multi-thread to accelerate the mesh generation process.

Generate 3D Shape

Generate a 3D shape including structure and details. (Section III.C.2)

1. Train parameters generator:

   cd gen_para
   python train.py train --cate_name chair_4
   

2. Train details generator for three views:

   cd gen_detail
   python train.py train --cate_name chair_4 --axis_name x --gpu 0
   python train.py train --cate_name chair_4 --axis_name y --gpu 0
   python train.py train --cate_name chair_4 --axis_name z --gpu 0
   

3. Generate 3D shape:

   cd gen_total_model
   python gen_total_model.py gen --cate_name chair_4                   # generate structure and details
   python gen_total_model.py gen_mesh_all --cate_name chair_4          # generate mesh 
   

   We provide the pretrained weights. You can directly generate the 3D shape after downloading the weights.

Interpolate 3D Shape

Interpolate a 3D shape between two 3D shapes. (Section IV.E)

Enter the interpolate_total_model directory.

1. Interpolate the 3D shape by interpolating the latent codes:

   python interpolate_total_model.py interpolate --cate_name chair_4 --id_1 25 --id_2 67
   python interpolate_total_model.py gen_mesh_all --cate_name chair_4 --id_1 25 --id_2 67 
   

2. Interpolate the 3D shape by interpolating the structure parameters:

   python interpolate_total_model.py interpolate --cate_name chair_4 --id_1 25 --id_2 67 --para_inter
   python interpolate_total_model.py gen_mesh_all --cate_name chair_4 --id_1 25 --id_2 67 --para_inter
   

Classify Points

Train a classifier to classify points into different categories. (Section IV.B.1)

Enter the classify directory.

1. Train the classifier:

   python classifier.py --mode train
   

2. Test the classifier:

   python classifier.py --mode test
   

   We provide the pretrained weights. You can directly run the test after downloading the weights.

3. Test the reconstruction with classifier:

   python classifier.py --mode test_reconstruct 
   

GUI Systerm

1. Install

   You need to install extra dependencies for the GUI system on the server (including all the dependencies above).

   pip install zerorpc msgpack msgpack-numpy
   

   You only need to install following packages on the client side (such as your PC) with Python 3.8:

   pip install imgui[pygame] PyOpenGL PyOpenGL_accelerate open3d pyvista zerorpc msgpack msgpack-numpy pyautogui pyyaml
   

2. Example Data

   You can download the example data app_example_data.zip from Baidu Yun(code: 2fe8) or NJU Box(code: psdt). Unzip the file and put the data directory in the app directory.

3. Run the server:

   cd server
   python server.py
   

4. Run the client:

   You need to change the IP address in the app/main.py (Line 50).

   cd app
   python main.py
   

   You can also run the app without server for only designing the 3D structure templates or annotations.

   python main.py --no_server
   

5. Usage

   See the video app.mp4 from Baidu Yun(code: 2fe8) or NJU Box(code: psdt).

License

MIT License

Acknowledgements

pointnet2 operation lib

pyvista_with_imgui

Citation

@ARTICLE{MA2025PSDT,
  author={Ma, Changfeng and Guo, Pengxiao and Yang, Shuangyu and Chen, Yinuo and Guo, Jie and Wang, Chongjun and Guo, Yanwen and Wang, Wenping},
  journal={IEEE Transactions on Visualization and Computer Graphics}, 
  title={Parameterize Structure With Differentiable Template for 3D Shape Generation}, 
  year={2025},
  volume={31},
  number={10},
  pages={8915-8927},
  doi={10.1109/TVCG.2025.3583987}
}