RGB and Thermal Feature-level Sensor Fusion (RGBT)

Download the FLIR pre-processed dataset here

ToDo List

• Finish literature review of both Object Detection and Fusion
• Find the conversion Matrix btw. RGB and IR Frame
• Prepare the FLIR Dataset for fusion
• Implement Scaled_Yolov4
• Test the RGB frames from FLIR Dataset on Scaled_Yolov4 (Baseline)
• Ablation Study for the Baseline
• Dive deeper into SPP, PAN, FPN, DCN, SAM, GIOU and other relevant papers for Fusion
• Design the Fusion Network
• Vanilla Fusion Training
• CBAM Fusion Training
• Entropy-based Channel Attention Module (EBAM)
• Entropy-based Spatial Attention Module
• Entropy-based Channel & Spatial Attention Module
• EBAM Fusion Training
• Day & Night Analysis
• Synthetic Fog Analysis
• TensorRT and Latency Results
• Installation and User Guidlines

Model	Test Size	PersonAP@.5test	BicycleAP@.5test	CarAP@.5test	OverallmAP@.5test	Num. of Param.
RGB Baseline	320	39.6%	50.4%	79.4%	56.6%	52.5
IR Baseline	320	49.6%	54.9%	84.4%	63.0%	52.5
Vanila Fusion	320	56.9%	56.7%	82.0%	65.2%	81.8
Fusion + CBAM	320	57.6%	60.5%	83.6%	67.2%	82.7
Fusion + EBAM_C	320	62.6%	65.9%	86.0%	71.5%	82.7%
RGBT	320	63.7%	67.1%	86.4%	72.4%	82.7
CFR_3	640	74.4%	57.7%	84.9%	72.3%	276
RGBT	640	80.1%	76.7%	91.8%	82.9%	82.7%

Repo Structure:

Datasets: A summary of all available dataset with IR-RGB image pair
FLIR_PP: Preprocessing of the FLIR dataset - The cross-labeling algorithm can be used by using pp.py
Fusion: The implementation of the RGBT fusion network
Related_Works: Literature Review
To Train use the train_org.py
To Test use the test_org.py

Citation

@INPROCEEDINGS{9827087,
  author={Vadidar, Sam and Kariminezhad, Ali and Mayr, Christian and Kloeker, Laurent and Eckstein, Lutz},
  booktitle={2022 IEEE Intelligent Vehicles Symposium (IV)}, 
  title={Robust Environment Perception for Automated Driving: A Unified Learning Pipeline for Visual-Infrared Object Detection}, 
  year={2022},
  volume={},
  number={},
  pages={367-374},
  keywords={Visualization;Intelligent vehicles;Fuses;Roads;Pipelines;Object detection;Thermal sensors},
  doi={10.1109/IV51971.2022.9827087}}