PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

PillarNeSt is a robust pillar-based 3D object detectors, which obtains 66.9%(SoTA without TTA/model ensemble) mAP and 71.6 % NDS on nuScenes benchmark.

Preparation

• Environments

Python == 3.6
CUDA == 11.1
pytorch == 1.9.0
mmcls == 0.22.1
mmcv-full == 1.4.2
mmdet == 2.20.0
mmsegmentation == 0.20.2
mmdet3d == 0.18.1

• Data
  Follow the mmdet3d to process the nuScenes dataset.

• Weights

Model weights are available at Google Drive and BaiduWangpan(1111).

Main Results

Results on nuScenes val set. (15e + 5e means the last 5 epochs should be trained without GTsample)

Config	mAP	NDS	Schedule	weights	weights
PillarNeSt-Tiny	58.8%	65.6%	15e+5e	Google Drive	Baidu
PillarNeSt-Small	61.7%	68.1%	15e+5e	Google Drive	Baidu
PillarNeSt-Base	63.2%	69.2%	15e+5e	Google Drive	Baidu
PillarNeSt-Large	64.3%	70.4%	18e+2e	Google Drive	Baidu

Results on nuScenes test set (without any TTA/model ensemble).

Config	mAP	NDS
PillarNeSt-Base	65.6 %	71.3%
PillarNeSt-Large	66.9%	71.6%

Citation

If you find PillarNeSt helpful in your research, please consider citing:

@article{mao2023pillarnest,
  title={PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection},
  author={Mao, Weixin and Wang, Tiancai and Zhang, Diankun and Yan, Junjie and Yoshie, Osamu},
  journal={arXiv preprint arXiv:2311.17770},
  year={2023}
}

---

TODO:

• weights on test set
• Backbone code
• add CenterPoint-Plus
• Small, Base, Large configs
• Upload weights to Baidu cloud

Contact

If you have any questions, feel free to open an issue or contact us at maoweixin@megvii.com (maowx2017@fuji.waseda.jp) or wangtiancai@megvii.com.

---

PS:

Recently, our team also conduct some explorations into the application of multi-modal large language model (MLLM) in the field of autonomous driving:

Adriver-I: A general world model for autonomous driving

@article{jia2023adriver,
  title={Adriver-i: A general world model for autonomous driving},
  author={Jia, Fan and Mao, Weixin and Liu, Yingfei and Zhao, Yucheng and Wen, Yuqing and Zhang, Chi and Zhang, Xiangyu and Wang, Tiancai},
  journal={arXiv preprint arXiv:2311.13549},
  year={2023}
}

PPS:

组内招收自驾大模型、世界模型和具身智能相关的实习生，详情咨询/简历投递：maoweixin@megvii.com