Repository of LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset accepted by CVPR 2024
[paper]/[Project page]
Complete raw data will be released soon.
Download the preprocessed data from BaiduYun (code: z64v). (Currently it only contains chair category, it will be fully released after the cleaning process) Put all the downloaded data under LASA, unzip the align_mat_all.zip mannually. You can choose to use the script ./process_scripts/unzip_all_data to unzip all the data in occ_data and other_data by following commands:
cd process_scripts
python unzip_all_data.py --unzip_occ --unzip_other
Run the following commands to generate augmented partial point cloud for synthetic dataset and LASA dataset
cd process_scripts
python augment_arkit_partial_point.py --cat arkit_chair arkit_stool ...
python augment_synthetic_partial_point.py --cat 03001627 future_chair ABO_chair future_stool ...
Run the following command to extract image features
cd process_scripts
bash dist_extract_vit.sh
Finally, run the following command to generate train/val splits, please check ./dataset/taxonomy for the sub-cateory definition:
cd process_scripts
python generate_split_for_arkit --cat arkit_chair arkit_stool ...
python generate_split_for_synthetic_data.py --cat 03001627 future_chair ABO_chair future_stool ...
Download the pretrained weight for chair from chair_checkpoint. (code:hlf9).
Put these folder under LASA/output.
The ae folder stores the VAE weight, dm folder stores the diffusion model trained on synthetic data.
finetune_dm folder stores the diffusion model finetuned on LASA dataset.
Run the following commands to evaluate and extract the mesh:
cd evaluation
bash dist_eval.sh
The category entries are the sub-category from arkit scenes, please see ./datasets/taxonomy.py about how they are defined. For example, if you want to evaluate on LASA's chair, category should contain both arkit_chair and arkit_stool. make sure the --ae-pth and --dm-pth entry points to the correct checkpoint path. If you are evaluating on LASA, make sure the --dm-pth points to the finetuned weight in the ./output/finetune_dm folder. The result will be saved under ./output_result.
Run the train_VAE.sh to train the VAE model. If you aims to train on one category, just specify one category from chair,
cabinet, table, sofa, bed, shelf. Inputting all will train on all categories. Makes sure to download and preprocess all
the required sub-category data. The sub-category arrangement can be found in ./datasets/taxonomy.py
After finish training the VAE model, run the following commands to pre-extract the VAE features for every object:
cd process_scripts
bash dist_export_triplane_features.sh
Then, we can start training the diffusion model on the synthetic dataset by running the train_diffusion.sh.
Finally, finetune the diffusion model on LASA dataset by running finetune_diffusion.sh.
Early stopping is used by mannualy stopping the training by 150 epochs and 500 epochs for training VAE model and diffusion model respetively. All experiments in the paper are conducted on 8 A100 GPUs with batch size = 22.
- Object Detection Code
- Code for Demo on both arkitscene and in the wild data
@article{liu2023lasa,
title={LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset},
author={Liu, Haolin and Ye, Chongjie and Nie, Yinyu and He, Yingfan and Han, Xiaoguang},
journal={arXiv preprint arXiv:2312.12418},
year={2023}
}