[CVPR 24] Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation

Sangyun Shin, Kaichen Zhou, Madhu Vankadari, Andrew Markham, Niki Trigoni
Cyber-Physical Systems Group, Department of Computer Science, University of Oxford

Abstract: Coarse-to-fine 3D instance segmentation methods show weak performances compared to recent Grouping-based, Kernel-based and Transformer-based methods. We argue that this is due to two limitations: 1) Instance size overestimation by axis-aligned bounding box(AABB) 2) False negative error accumulation from inaccurate box to the refinement phase. In this work, we introduce Spherical Mask, a novel coarse-to-fine approach based on spherical representation, overcoming those two limitations with several benefits. Specifically, our coarse detection estimates each instance with a 3D polygon using a center and radial distance predictions, which avoids excessive size estimation of AABB. To cut the error propagation in the existing coarse-to-fine approaches, we virtually migrate points based on the polygon, allowing all foreground points, including false negatives, to be refined. During inference, the proposal and point migration modules run in parallel and are assembled to form binary masks of instances. We also introduce two margin-based losses for the point migration to enforce corrections for the false positives/negatives and cohesion of foreground points, significantly improving the performance. Experimental results from three datasets, such as ScanNetV2, S3DIS, and STPLS3D, show that our proposed method outperforms existing works, demonstrating the effectiveness of the new instance representation with spherical coordinates.

Details of the model architecture and experimental results can be found in our paper.

Quick Demo 🔥

ScanNetv2

Dataset	AP	AP_50	Config	Checkpoint
ScanNet val	62.6	81.9	config	checkpoint

• December, 2023: Spherical Mask achieves state-of-the-art in ScanNet-V2 3D instance segmentation. [Link] [Screenshot]

For the best training result, we recommend initializing the encoder with the pretrained-weights checkpoint(Download) from ISBNet. After downloading the pre-trained weights, please specify the path in configs/scannetv2/spherical_mask.yaml

# train 
python tools/train.py configs/scannetv2/spherical_mask.yaml --trainall --exp_name defaults
# test
python tools/test.py configs/scannetv2/spherical_mask.yaml --ckpt path_to_ckpt.pth

The code has been tested using torch==1.12.1 and cuda==11.3 on Ubuntu 20.04.

Please CITE our paper if you found this repository helpful for producing publishable results or incorporating it into other software.

@inproceedings{shin2024spherical,
 author={Sangyun Shin, Kaichen Zhou, Madhu Vankadari, Andrew Markham, Niki Trigoni},
 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
 title={Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation},
 year= {2024}
}

Datasets 💾

• ScanNetV2

Acknowledgements 👏

This repo is built upon ISBNet, SpConv, DyCo3D, SSTNet, and SoftGroup.

Contacts 📧

If you have any questions or suggestions about this repo, please feel free to contact me (kimshin812@gmail.com or sangyun.shin@cs.ox.ac.uk).