This repository provides Docker support for Sparse4D, complete with Dockerfiles and necessary modifications for seamless integration.
The expected folder structure for this setup is as follows:
Workspace/
│
├── data/ # Data directory
│ ├── nuscenes/ # Example dataset
│ ├── ... # Other data folders
│
└── Sparse4D/ # Sparse4D source code
├── ...
├── docker/
├── docker-compose.yml
├── ...
You can modify the paths as needed in the ./docker-compose.yml file.
The Docker Compose file is designed to support multiple modes of operation, such as data preparation, training, testing, or launching an interactive bash shell.
To get started, run the following command in your terminal:
SPARSE4D_MODE=bash docker-compose upYou can change the operation mode by modifying the SPARSE4D_MODE environment variable. Available modes include:
train: For training models.test: For testing models.data_prep: For preparing data.bash: To launch an interactive bash shell.
For detailed mode configurations, refer to ./docker/entrypoint.sh.
- The pre-installed NCCL (NVIDIA Collective Communications Library) version is not supported under Windows Subsystem for Linux (WSL). As a result, only single GPU usage is feasible.
- A potential workaround is to rebuild PyTorch with a newer NCCL version (greater than 2.10) to resolve this issue.
【2023/11/21】 The paper of Sparse4Dv3 has been published.
Sparse4Dv3 is about to be released, featuring stronger detection performance and end-to-end tracking capabilities.
State-of-the-Art Performance of Sparse4Dv2 in the nuScenes Benchmark for Online Models.
These experiments were conducted using 8 RTX 3090 GPUs with 24 GB memory.
| model | backbone | pretrain | img size | Epoch | Traning | FPS | NDS | mAP | config | ckpt | log |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sparse4D-T4 | Res101 | FCOS3D | 640x1600 | 24 | 2Day5H | 2.9 | 0.5438 | 0.4409 | cfg | ckpt | log |
| Sparse4Dv2 | Res50 | ImageNet | 256x704 | 100 | 15H | 20.3 | 0.5384 | 0.4392 | cfg | ckpt | log |
| Sparse4Dv2 | Res101 | nuImage | 512x1408 | 100 | 2Day | 8.4 | 0.5939 | 0.5051 | - | - | - |
| model | backbone | img size | NDS | mAP | mATE | mASE | mAOE | mAVE | mAAE |
|---|---|---|---|---|---|---|---|---|---|
| Sparse4D-T4 | Vov-99 | 640x1600 | 0.595 | 0.511 | 0.533 | 0.263 | 0.369 | 0.317 | 0.124 |
| Sparse4Dv2 | Vov-99 | 640x1600 | 0.638 | 0.556 | 0.462 | 0.238 | 0.328 | 0.264 | 0.115 |
Install requirements.
pip install -r requirements.txt
cd projects/mmdet3d_plugin/ops
python setup.py developDownload nuScenes dataset, pretrain checkpoint(fcos3d.pth ResNet101), pkl files(nuscenes_infos_trainval_with_inds.pkl) and init anchor centers(nuscenes_kmeans900.npy). Adjust the directory structure as follows:
Sparse4D
├── data
│ ├── nuscenes
│ │ ├── maps
│ │ ├── lidarseg
│ │ ├── samples
│ │ ├── sweeps
│ │ ├── v1.0-mini
│ │ ├── v1.0-test
| | └── v1.0-trainval
│ ├── nuscenes_cam
│ │ ├── nuscenes_infos_test.pkl
│ │ ├── nuscenes_infos_train.pkl
│ │ ├── nuscenes_infos_val.pkl
│ │ └── nuscenes_infos_trainval_with_inds.pkl
├── projects
│ ├── configs
│ │ ├── default_runtime.py
│ │ ├── sparse4d_r101_H1.py
│ │ ├── sparse4d_r101_H4.py
│ │ └── ...
│ └── mmdet3d_plugin
│ ├── apis
│ ├── core
│ ├── datasets
│ └── models
├── tools
│ ├── dist_test.sh
│ ├── dist_train.sh
│ ├── test.py
│ └── train.py
├── local_test.sh
├── local_train.sh
├── fcos3d.pth
└── nuscenes_kmeans900.npyTrain with config_name.py.
bash local_train.sh config_nameTest checkpoint_file with config_name.py.
bash local_test.sh config_name checkpoint_file @misc{2305.14018,
Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
Title = {Sparse4D v2: Recurrent Temporal Fusion with Sparse Model},
Year = {2023},
Eprint = {arXiv:2305.14018},
}
@misc{2211.10581,
Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
Title = {Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion},
Year = {2022},
Eprint = {arXiv:2211.10581},
}