Cross Modality Knowledge Distillation for Robust VRU detection in low light and adverse weather conditions

Project Overview

This repository is part of a larger project focused on utilizing knowledge distillation to transfer features from various modalities to a single modality using any type of deep learning model. The approach is both sensor and model agnostic. In this specific use case, we explored two knowledge distillation techniques to enhance the performance and robustness of object detectors in adverse weather and low-light conditions, relying solely on an RGB camera without adding extra sensors or cameras during inference.

Paper: Cross Modality Knowledge Distillation for Robust Pedestrian Detection in Low Light and Adverse Weather Conditions

This implementation is based on Faster R-CNN with ResNet50-FPN backone in Pytorch using Seeing Through Fog dataset.

Trained and tested with python: 3.9.7, Ubuntu:18.04.5 LTS, Cuda: 11.2, Pytorch:1.11, GPU: Geforce RTX 3090

Usage

• Install PyTorch.
• Download the data from here and extract the ZIP file in data/ folder.
• Download the trained teacher network or train it by running this comment. The teacher network is trained using both RGB images and 3 Gated slices in the dataset.

python train_teacher.py

• train Cross Modality Knowledge Distillation (CMKD) method based on Mean Squred Error (MSE) of backnone features by running this comment:

python train_cmkd_mse.py

• train Cross Modality Knowledge Distillation (CMKD) method based on adversarial training of backnone features by running this comment:

python train_cmkd_adv.py

• The trained network can be evaluated using val and test sets by running this comment:

python evaluate.py

• The baseline network can be trained by running this comment. Baseline is trained using only RGB images without CMKD.

python train_baseline.py

• Visual detection examples can be seen by running this comment:

visual_detect.py

Results & Pretrained Weights

Model	COCO mAP val set	COCO mAP test set	Trained model
Teacher	25.8	27.5	download
Baseline	22.5	24.2	download
CMKD-MSE	23.6	25.4	download
CMKD-Adv	24.2	26.0	download

Visual detection examples

knowledge distillation successfully detects pedestrians that baseline does not detect.

knowledge distillation removes the false positive that baseline has.

Citation

If you find this code useful, please consider citing:

@INPROCEEDINGS{10095353,
  author={Hnewa, Mazin and Rahimpour, Alireza and Miller, Justin and Upadhyay, Devesh and Radha, Hayder},
  booktitle={ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
  title={Cross Modality Knowledge Distillation for Robust Pedestrian Detection in Low Light and Adverse Weather Conditions}, 
  year={2023},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/ICASSP49357.2023.10095353}}
}