3DCoMPaT++: The Winning Solution 🏆 for 3D Compositional Part and Material Segmentation challenge.

Summary

• Introduction
• Getting started
• Model Performance
• Acknowledgments
• Citation

📚 Introduction

This repo contains the winning model 🏆 of both coarse and fine-grained tracks for 3DCoMPaT Grounded CoMPaT Recognition Challenge organized by C3DV CVPR 2023 workshop. Built on top of the original 3DCoMPaT-v2 challenge codebase, the model uses PointNet++ as a backbone and expands training to 6D (rgb). This 3D point clouds segmentation model achieves accuracy of material segmentation 98% (from 46%) and part segmentation 97% (from 85%). This work also explores multimodal 2D/ 3D training PointNet++ with 2D segmentation logits from the given pretrained SegFormer as well as top-3 2D segmentation label which is more effective in increasing accuracy (1-2%).

We plan to further extend the work: stay tuned! 🔥

🚀 Getting started

Train the model

• To train the part segmentation model, run python train_partseg.py --data_name=coarse --batch_size=128.
• To train the mat segmentation model, run python train_matseg.py --data_name=coarse --batch_size=128.
• The pre-trained model will be uploaded soon: stay tuned!

Inference

• To run inference of shape classification, part and material segmentation together, python predict.py --data_type=coarse --n_comp=10 --batch_size=160 --split=test.

📊 Model Performance

• Model performance comparison on shape-aware part and material segmentation. The first three rows (partseg) is measured by organizers as reported here.

Performance on validation dataset

Coarse-grained

Model	PartSeg Accuracy	PartSeg mIOU	MatSeg Accuracy	MatSeg mIOU	ckpt
PointNet2	84.72	77.98	46.62	38.59	gdrive
Curvenet	86.01	80.64	N/A	N/A	gdrive
PointNeXt	94.17	86.80	N/A	N/A	gdrive
PointNet2 6D (ours)	97.05	87.74	98.65	96.24	TODO

Fine-grained

Model	PartSeg Accuracy	PartSeg mIOU	MatSeg Accuracy	MatSeg mIOU	ckpt
PointNet2	71.09	80.01	TODO	TODO	gdrive
Curvenet	72.49	81.37	N/A	N/A	gdrive
PointNeXt	82.07	83.92	N/A	N/A	gdrive
PointNet2 6D (ours)	86.33	77.62	98.86	94.42	TODO

Challenge metrics compared with the baseline

🙏 Acknowledgments

⚙️ Thank you Habib Slim, Professor Mohamed Elhoseiny, and the challenge organizer for making the great challenge and very useful codebase for us to experiment on top very easily.

Citation

If you use their 3DCoMPaT++ dataset, please cite the two following references:

@article{slim2023_3dcompatplus,
    title={3DCoMPaT++: An improved Large-scale 3D Vision Dataset
    for Compositional Recognition},
    author={Habib Slim, Xiang Li, Yuchen Li,
    Mahmoud Ahmed, Mohamed Ayman, Ujjwal Upadhyay
    Ahmed Abdelreheem, Arpit Prajapati,
    Suhail Pothigara, Peter Wonka, Mohamed Elhoseiny},
    year={2023}
}

@article{li2022_3dcompat,
    title={3D CoMPaT: Composition of Materials on Parts of 3D Things},
    author={Yuchen Li, Ujjwal Upadhyay, Habib Slim,
    Ahmed Abdelreheem, Arpit Prajapati,
    Suhail Pothigara, Peter Wonka, Mohamed Elhoseiny},
    journal = {ECCV},
    year={2022}
}

This repository is owned and maintained by Cattalyya Nuengsigkapian.

References

1. [Li et al., 2022] - 3DCoMPaT: Composition of Materials on Parts of 3D Things.
2. [Xie et al., 2021] - SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.
3. [He et al., 2015] - Deep Residual Learning for Image Recognition.