This repo is the official project repository of the paper [SFPNet] (ECCV 2024) and dataset [S.MID].
SFPNet: For the backbone of LiDAR segmentation task, the three main challenges it needs to address are sparsity, large scale, and non-uniform changes in point cloud density. Previous works introduced inductive bias (special partition or special window and position encoding) to a single type of LiDAR (usually mechanical spinning LiDAR) to solve the above three challenges . This can limit model generalizability to other kinds of LiDAR and make hyperparameter tuning more complex. Therefore, we propose the pipeline SFPNet with favorable properties (See more in our paper) to address the three main challenges in various type of LiDAR point cloud data while avoiding the introduction of special inductive bias. Our core module SFPM can replace window transformer block.
Comparison of field of view:
Comparison of cumulative point clouds:
S.MID: We used an industrial robot (equipped with Livox Mid-360) to collect a total of 38904 frames of hybrid-solid LiDAR data in different substations. We annotated 25 categories under professional guidance and merged them into 14 classes for single-frame segmentation task.
Examples of labeled cumulative point clouds in S.MID:
The S.MID dataset is published under CC BY-NC-SA 4.0 license, which means that anyone can use this dataset for non-commercial research purposes.
Download and create environment.
git clone https://github.com/Cavendish518/SFPNet.git
conda create --name sfpn python=3.9
conda activate sfpn
Install dependencies.
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch_scatter==2.0.9
pip install torch_geometric==1.7.2
pip install spconv-cu114==2.1.21
pip install torch_sparse==0.6.12 cumm-cu114==0.2.8 torch_cluster==1.5.9
pip install tensorboard timm termcolor tensorboardX
pip install numpy==1.25.0
Demo for S.MID:
python demo_SMID.py --config config/demo/Mid_demo.yaml
Model weights library
| dataset | Val mIoU (tta) | Download |
|---|---|---|
| S.MID_beta_v1.2 | 73.4 (demo version) | S.MID Weight1 |
| S.MID_beta_v1.2 | 71.9 (original paper version) | S.MID Weight2 |
| nuScenes | 80.1 | nuScenes Weight |
| SemanticKITTI | 69.2 | SemanticKITTI Weight |
To facilitate the use of LiDAR semantic segmentation in downstream tasks, we provide ROS tools.
Please note that our work focuses on the representational capabilities of the network design itself. Therefore, related powerful training techniques (novel data augmentation, post-processing and distillation) have not been used in the experiments. But these are important parts of research. So we encourage you to try related techniques. We will be happy if you train better model weights. If you are willing to make the model weights and codes public, please contact us.
Download S.MID beta version 1.2 from [S.MID]. Details about dataset can be found in the webpage and our paper. An optimized and denoised version(v1.3) is coming soon.
For constructing the code for dataloader, you can refer to the implementation in the demo.
Additional dependencies for dataset tools.
pip install matplotlib
pip install open3d
- Release model code of SFPNet;
- Release S.MID and instruction in our dataset webpage;
- Release code of Demo (validation);
- Release dataset tools;
- Release model weights;
- S.MID
- nuScenes
- SemanticKITTI
- Release code of training;
- Release ROS tools.
We would like to thank all the pioneers SemanticKITTI, FocalNet, SphereFormer, Cylinder3D and nuScenes.
If your like our paper, codes or dataset, please cite us and give this repo a star.
@article{wang2024sfpnet,
title={SFPNet: Sparse Focal Point Network for Semantic Segmentation on General LiDAR Point Clouds},
author={Wang, Yanbo and Zhao, Wentao and Cao, Chuan and Deng, Tianchen and Wang, Jingchuan and Chen, Weidong},
journal={arXiv preprint arXiv:2407.11569},
year={2024}
}