Distributed Cooperative Sensing: Facilitates distributed deep fusion for hybrid Radar/Camera/Lidar data and multi-station cooperative sensing. The architecture of our proposed AI processing framework for 2D3DFusion is shown here:
The document for 3D detection model design and Kitti and Waymo data training: https://mytutorial-lkk.readthedocs.io/en/latest/mydetector3d.html
The document for 3D detection based on V2X cooperative Lidar sensing data: https://mytutorial-lkk.readthedocs.io/en/latest/3DV2X.html
Nuscence dataset and BEV transform based on Lift Splat is available on: https://mytutorial-lkk.readthedocs.io/en/latest/nuscenes.html
Clone this repository, install this package (need NVIDIA CUDA environment)
python setup.py developInstall the SparseConv library from spconv and numba:
pip install spconv-cu117 #pip install spconv-cu118
pip install numbabuild additional cuda ops libraries via
(mycondapy39) [010796032@cs001 3DDepth]$ module load cuda-11.8.0-gcc-11.2.0-5tlywx3 #should match pytorch cuda version
(mycondapy39) [010796032@cs001 3DDepth]$ python cudasetup.py build_ext --inplace
pip install nuscenes-devkit #required by nuscenes dataset
pip install efficientnet_pytorch==0.7.0 #required by lss
pip install pynvml
pip install nvidia-ml-py3 #required by import nvidia_smi
pip3 install --upgrade pyside2 pyqt5 #qt.qpa.plugin: Could not load the Qt platform plugin "xcb"
pip install kornia #required by BEVFusion
$ module load gcc/11.2.0
$ python cudasetup.py build_ext --inplaceInstall 'mayavi' (ref: https://docs.enthought.com/mayavi/mayavi/installation.html) and open3d (ref: http://www.open3d.org/docs/release/getting_started.html) for 3d point cloud visualization
conda create --name mycondapy311 python=3.11
conda activate mycondapy311
pip install pyqt6
#test pyqt6: sdrpysim/testpyqt6.py
pip install pyqtgraph
#Successfully installed numpy-1.26.1 pyqtgraph-0.13.3
#import pyqtgraph as pg
#test pyqtgraph: sdrpysim\pyqt6qtgraphtest.py
pip install matplotlib #conda install matplotlib will install pyqt5
#Successfully installed contourpy-1.2.0 cycler-0.12.1 fonttools-4.44.0 kiwisolver-1.4.5 matplotlib-3.8.1 packaging-23.2 pillow-10.1.0 pyparsing-3.1.1 python-dateutil-2.8.2 six-1.16.0
pip install opencv-python-headless
pip install mayavi
#pip3 install PySide6 #will cause pyqt6 not working, but mayavi needs PySide6
pip install pyqt5 #needed by mayavi and matplotlib
conda install -c conda-forge jupyterlab
python VisUtils/testmayavi.py #test mayavi installation
pip install open3d #does not support python3.11, only 3.7-3.10
#install development version of open3d: http://www.open3d.org/docs/latest/getting_started.html
pip install -U --trusted-host www.open3d.org -f http://www.open3d.org/docs/latest/getting_started.html open3d
# Verify installation
python -c "import open3d as o3d; print(o3d.__version__)"
# Open3D CLI
open3d example visualization/draw
python VisUtils/testopen3d.py #test open3d installationmmdetection3d installation based on installation
pip install -U openmim
mim install mmengine
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0'
(mycondapy310) PS D:\Developer> &'C:\Program Files\Git\bin\git.exe' --version
(mycondapy310) PS D:\Developer> &'C:\Program Files\Git\bin\git.exe' clone https://github.com/open-mmlab/mmdetection3d.git
(mycondapy310) PS D:\Developer\mmdetection3d> pip install -v -e .
>>> import mmdet3d
>>> print(mmdet3d.__version__)
1.4.0
pip install cumm-cu118
pip install spconv-cu118
#test installation
(mycondapy310) PS D:\Developer\mmdetection3d> mim download mmdet3d --config pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car --dest .
(mycondapy310) PS D:\Developer\mmdetection3d> python demo/pcd_demo.py demo/data/kitti/000008.bin pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth
cat .\outputs\preds\000008.json
{"labels_3d": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], "scores_3d": (mycondapy310) [010796032@cs001 3DDepth]$ python ./mydetector3d/tools/mytrain.pyCheck kittidata for detailed information of Kitti dataset.
Check waymodata for detailed information of Waymo dataset.