Skip to content

Latest commit

 

History

History
93 lines (73 loc) · 3.41 KB

README.md

File metadata and controls

93 lines (73 loc) · 3.41 KB
Raw

Quantization for BEVFusion

BEVFusion's SparseConvolution module uses the mmdet3d/spconv

QAT Workflow

  • Insert Q&DQ nodes to get fake-quant pytorch model
  • PTQ calibration
  • QAT Training

Notes

  • FuseBn can bring better performance to the model forward, so this operation needs to be completed before model calibration.

  • The Add and Concat layers have multiple inputs，need to use the same quantizer for all inputs, thus reducing the occurrence of Reformat.

  • The quantization of some layers will cause a significant drop in mAP, so it is necessary to disable the quantization of these layers after calibration.


Usage

1. Configuring the bevfusion runtime environment

  • Here is the official configuration guide.
  • Setup the bevfusion runtime environment:
# build image from dockerfile
cd CUDA-BEVFusion/bevfusion/docker
docker build . -t bevfusion

# creating containers and mapping volumes
nvidia-docker run -it -v `pwd`/../../../:/Lidar_AI_Solution \
     -v /path/to/nuScenes:/data \
     --shm-size 16g bevfusion   

# install python dependency libraries
cd /Lidar_AI_Solution/CUDA-BEVFusion
pip install -r tool/requirements.txt

# install bevfusion
cd bevfusion
python setup.py develop

2. Download model.zip and nuScenes-example-data.zip

# download models and datas to CUDA-BEVFusion
cd CUDA-BEVFusion

# unzip models and datas
apt install unzip
unzip model.zip
unzip nuScenes-example-data.zip

# copy yaml to bevfusion
cp -r configs bevfusion

3. Export INT8 model

python qat/export-camera.py --ckpt=model/resnet50int8/bevfusion_ptq.pth
python qat/export-transfuser.py --ckpt=model/resnet50int8/bevfusion_ptq.pth
python qat/export-scn.py --ckpt=model/resnet50int8/bevfusion_ptq.pth --save=qat/onnx_int8/lidar.backbone.onnx

4. Export FP16 model

python qat/export-camera.py --ckpt=model/resnet50int8/bevfusion_ptq.pth --fp16
python qat/export-transfuser.py --ckpt=model/resnet50int8/bevfusion_ptq.pth --fp16
python qat/export-scn.py --ckpt=model/resnet50int8/bevfusion_ptq.pth --save=qat/onnx_fp16/lidar.backbone.onnx

5. Generate PTQ model

  • This code uses the nuScenes Dataset. You need to download it in order to run PTQ.
    • You can follow the tips here to prepare the data.
python qat/ptq.py --config=bevfusion/configs/nuscenes/det/transfusion/secfpn/camera+lidar/resnet50/convfuser.yaml --ckpt=model/resnet50/bevfusion-det.pth --calibrate_batch 300

6. Evaluating mAP with PyBEV

cd CUDA-BEVFusion
cp qat/test-mAP-for-cuda.py bevfusion/tools

cd bevfusion
mkdir data
ln -s /path/to/nuScenes data/nuscenes

python tools/test-mAP-for-cuda.py

Workflow in QAT Mode

The performance metrics of the PTQ INT8 model are already very close to fp16, and work on the QAT part will follow.