SparseSED: A New Paradigm for Multi-View 3D Object Detection Based on 2D and 3D Supervised Fusion in Autonomous Driving Systems

Overall Architecture

The framework synergistically integrates 2D and 3D annotations for enhanced supervision. Our approach extends conventional sparse models by introducing three novel components: (1) ​multi-view ROIs prediction, which maps 3D anchors onto multi-view image planes to predict regions of interest (ROIs); (2) ​multi-view instance segmentation, generating pixel-accurate masks corresponding to predicted ROIs; and (3) ​cross-modal feature alignment, which fuses geometrically consistent 2D segmentation features with their associated 3D anchors. During training, 2D instance segmentation masks provide auxiliary supervision alongside standard 3D annotations, enabling the model to exploit both structural and semantic cues.

What we provide

(1) sparseSED model

(2) how to train sparseSED

(3) how to val sparseSED

(4) the trained model of sparseSED

(5) the val result of sparseSED

(6) the additional auxiliary supervision dataset nuScenesSeg

(7) the spares4d v3 model, train and val of spares4d v3, trained model and val result of sparse4d v3 is also included in the project.

nuScenes Benchmark

Results on Validation Split

These experiments were conducted using 1 RTX 4090 GPU with 24 GB memory. |model | backbone | img size | Epoch | FPS | NDS | mAP | config | | :----: | :---: | :---: | :---: | :---: | :---:| :---:|:---:|:---: | :---: | |Sparse4Dv3|Res50|256x704| 120 | 19.8 |0.5637|0.4646|sparse4dv3_temporal_r50_1x8_bs6_256x704.py| |Sparse4Dv3|Res101|512x1408| 120 | 19.2 |0.571|0.467|sparseSED_temporal_r50_1x8_bs6_256x704.py|

Quick Start

Please install all the packages we need!

Before training, just prepare cuda environment and nuScenes dataset as Sparse4D v3.

Segmentation dataset is made by ourselves, it is available at nuScenesSeg. This dataset is necessary in training, please download it.

(1) For training:

Just run train_sparseSED.py if you want to train sparseSED.

The network will be saved in "work_dirs/--your config name--/"

The configs of Sparse4D v3 and SparseSED are in "configs/". If you want to change the configs, you can just edit them.

(2) For testing:

We give a trained network in "iter_468800.pth", you can run test_sparseSED.py to test it. The test result will be saved in "eval/--your config name--/".

You can also use test_sparseSED.py to test your own trained model, but remember change the network path and config filename in test_sparseSED.py.

We have also give the test result in iter_468800.

Acknowledgement

• Sparse4D v3
• BEVFormer
• DETR3D
• mmdet3d
• SOLOFusion
• StreamPETR