CVPR 2024 | 3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling

Check out the project demo here: 3DSFLabelling-Page

The code is gradually being released, please be patient.

[poster coming soon] [video coming soon]

Description	Simple Example of the Auto-Labelling
The proposed 3D scene flow pseudo-auto-labelling framework. Given point clouds and initial bounding boxes, both global and local motion parameters are iteratively optimized. Diverse motion patterns are augmented by randomly adjusting these motion parameters, thereby creating a diverse and realistic set of motion labels for the training of flow estimation models.

Highlights

🔥 GENERATE FULLY ALIGNED INTER-FRAME POINT CLOUDS

We propose a new framework for the automatic labelling of 3D scene flow pseudo-labels, significantly enhancing the accuracy of current scene flow estimation models, and effectively addressing the scarcity of 3D flow labels in autonomous driving.

:fire

🌟 Plug-and-play & Novel motion augmentation

We propose a universal 3D box optimization method with multiple motion attributes. Building upon this, we further introduce a plug-and-play 3D scene flow augmentation module with global-local motions and motion status. This allows for flexible motion adjustment of ego-motion and dynamic environments, setting a new benchmark for scene flow data augmentation.

News

• [2024/4] 🔥 Open source example model (GMSF, MSBRN and FLOT), Training and evaluation code.
• [2024/4] 🔥 Open sourced a new data set with 3D scene flow GT, The data comes from KITTI Odometry Dataset, Argoverse Dataset and nuScenes Dataset respectively.
• [2024/4] 🔥 Open sourced the code for the 3D Scene Flow Label Generation
• [2024/03] 3DSFLabelling code and models initially released.
• [2024/02] 3DSFLabelling is accepted by CVPR 2024.
• [2024/02] 3DSFLabelling paper released.

TODO List

Still in progress:

• Datasets are easier to use.
• The validity of the generated labels is verified on motion segmentation and LiDAR odometry.
• Readability optimization of configuration files and data reading code sections.

Table of Contents

1. Results and Model Zoo
2. License and Citation
3. Comparative Results

Results and Model Zoo

Method	Dataset	Pre-trained Model	EPE3D
GMSF	lidarKITTI	gmsf_lidarKITTI_epe0.008.pth	0.008
GMSF	Argoverse	gmsf_argoverse_epe0.013.pth	0.013
GMSF	nuScenes	gmsf_nuScene_epe0.018.pth	0.018
FLOT	lidarKITTI	flot_lidarKITTI_epe0.018.tar	0.018
FLOT	Argoverse	flot_argoverse_epe0.043.tar	0.043
FLOT	nuScenes	flot_nuScenes_epe0.061.tar	0.061
MSBRN	lidarKITTI	msbrn_lidarKITTI_epe0.011.pth	0.0110
MSBRN	Argoverse	msbrn_argoverse_epe0.017.pth	0.017
MSBRN	nuScenes	msbrn_nuScenes_epe0.076.pth	0.076

Comparative results

The comparative results between our method and baseline. "↑" signifies accuracy enhancement. In real-world LiDAR scenarios, our method markedly improves the 3D flow estimation accuracy across three datasets on the three baselines. This demonstrates that the proposed pseudo-auto-labelling framework can substantially boost the accuracy of existing methods, even without the need for ground truth.

Dataset	Method	EPE3D↓	Acc3DS↑	Acc3DR↑
	FLOT [1]	0.6532	0.1554	0.3130
	FLOT+3DSFlabelling	0.0189 ↑97.1%	0.9666	0.9792
	MSBRN [2]	0.0139	0.9752	0.9847
LiDAR KITTI	MSBRN+3DSFlabelling	0.0123 ↑11.5%	0.9797	0.9868
	GMSF [3]	0.1900	0.2962	0.5502
	GMSF+3DSFlabelling	0.0078 ↑95.8%	0.9924	0.9947

Dataset	Method	EPE3D↓	Acc3DS↑	Acc3DR↑
	FLOT [1]	0.2491	0.0946	0.3126
	FLOT+3DSFlabelling	0.0107 ↑95.7%	0.9711	0.9862
Argoverse	MSBRN [2]	0.8691	0.2432	0.2854
	MSBRN+3DSFlabelling	0.0150 ↑98.3%	0.9482	0.9601
	GMSF [3]	7.2776	0.0036	0.0144
	GMSF+3DSFlabelling	0.0093 ↑99.9%	0.9780	0.9880

Dataset	Method	EPE3D↓	Acc3DS↑	Acc3DR↑
	FLOT [1]	0.4858	0.0821	0.2669
	FLOT+3DSFlabelling	0.0554 ↑88.6%	0.7601	0.8909
nuScenes	MSBRN [2]	0.6137	0.2354	0.2924
	MSBRN+3DSFlabelling	0.0235 ↑96.2%	0.9413	0.9604
	GMSF [3]	9.4231	0.0034	0.0086
	GMSF+3DSFlabelling	0.0185 ↑99.8%	0.9534	0.9713

[1] Puy G, Boulch A, Marlet R. Flot: Scene flow on point clouds guided by optimal transport[C]//European conference on computer vision. Cham: Springer International Publishing, 2020: 527-544.
[2] Cheng W, Ko J H. Multi-scale bidirectional recurrent network with hybrid correlation for point cloud based scene flow estimation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 10041-10050.
[3] Zhang Y, Edstedt J, Wandt B, et al. Gmsf: Global matching scene flow[J]. Advances in Neural Information Processing Systems, 2024, 36.

License and Citation

All assets and code are under the Apache 2.0 license unless specified otherwise.

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{yang2023vidar,
  title={3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling},
  author={Jiang, Chaokang and Wang, Guangming and Liu, Jiuming and Wang, Hesheng and Ma, Zhuang and Liu, Zhenqiang and Liang, Zhujin and Shan, Yi and Du, Dalong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}