English | 简体中文

4D Contrastive Superflows are Dense 3D Representation Learners

Xiang Xu^1,*,    Lingdong Kong^2,3,*,    Hui Shuai⁴,    Wenwei Zhang²,   
Liang Pan²,    Kai Chen²,    Ziwei Liu⁵,    Qingshan Liu^4
1Nanjing University of Aeronautics and Astronautics    ²Shanghai AI Laboratory    ³National University of Singapore    ⁴Nanjing University of Posts and Telecommunications    ⁵S-Lab, Nanyang Technological University   

About

SuperFlow is introduced to harness consecutive LiDAR-camera pairs for establishing spatiotemporal pretraining objectives. It stands out by integrating two key designs: 1) a dense-to-sparse consistency regularization, which promotes insensitivity to point cloud density variations during feature learning, and 2) a flow-based contrastive learning module, carefully crafted to extract meaningful temporal cues from readily available sensor calibrations.

Updates

• [2024.07] - Our paper is accepted by ECCV.

Outline

• Installation
• Data Preparation
• Getting Started
• Main Results
• License
• Citation
• Acknowledgements

⚙️ Installation

For details related to installation and environment setups, kindly refer to INSTALL.md.

♨️ Data Preparation

Kindly refer to DATA_PREPAER.md for the details to prepare the datasets.

🚀 Getting Started

To learn more usage about this codebase, kindly refer to GET_STARTED.md.

📊 Main Results

Comparisons of state-of-the-art pretraining methods

Domain generalization study

Out-of-distribution 3D robustness study

License

This work is under the Apache 2.0 license.

Citation

If you find this work helpful for your research, please kindly consider citing our paper:

@inproceedings{xu2024superflow,
    title = {4D Contrastive Superflows are Dense 3D Representation Learners},
    author = {Xu, Xiang and Kong, Lingdong and Shuai, Hui and Zhang, Wenwei and Pan, Liang and Chen, Kai and Liu, Ziwei and Liu, Qingshan},
    booktitle = {European Conference on Computer Vision},
    year = {2024}
}

Acknowledgements

This work is developed based on the MMDetection3D codebase.

MMDetection3D is an open-source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

We acknowledge the use of the following public resources during the couuse of this work: ¹nuScenes, ²nuScenes-devkit, ³SemanticKITTI, ⁴SemanticKITTI-API, , ⁵WaymoOpenDataset, ⁶Synth4D, ⁷ScribbleKITTI, ⁸RELLIS-3D, ⁹SemanticPOSS, ¹⁰SemanticSTF, ¹¹SynthLiDAR, ¹²DAPS-3D, ¹³Robo3D, ¹⁴SLidR, ¹⁵DINOv2, ¹⁶Segment-Any-Point-Cloud, ¹⁷OpenSeeD, ¹⁸torchsparse. 💟