E2E-MFD

E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

The code is based on MMdetection 2.26.0, MMrotate 0.3.4 and MMCV-full 1.7.2. We modify its data loading, related classes, and functions. We revise the MMdetection and MMrotate to a multi-modal oriented detection framework to facilitate Multimodal Object Detection.

Overview

Getting Started

Installation

ref : mmrotate installation and mmdetection installation

Step 1: Clone the E2E-MFD repository:

To get started, first clone the E2E-MFD repository and navigate to the project directory:

git clone *****
cd *****

Step 2: Environment Setup:

E2E-MFD recommends setting up a conda environment and installing dependencies via pip. Use the following commands to set up your environment:

Create and activate a new conda environment

conda create -n E2E-MFD python=3.9.17
conda activate E2E-MFD

If you develop and run mmrotate directly, install it from source

pip install -v -e .

Install Dependencies

pip install -r requirements.txt

Prepare the dataset DroneVehicle

DroneVehicle is a publicly available dataset.

you can download the dataset at baiduyun with train (code:ngar) and test (code:tqwc).

root
├── DroneVehicle
│   ├── train
│   │   ├── rgb
│   │   │   ├── images
│   │   │   ├── labels
│   │   ├── ir
│   │   │   ├── images
│   │   │   ├── labels
│   ├── test
│   │   ├── rgb
│   │   │   ├── images
│   │   │   ├── labels
│   │   ├── ir
│   │   │   ├── images
│   │   │   ├── labels

Begin to train and test

Use the config file with this.

python ./tools/train.py
python ./tools/test.py

Generate fusion images

python ./tools/generate_fusion_image.py

Result

DroneVehicle weights
DroneVehicle logs

Future

The paper is under review, and this code repository is complete for rotating object detection, we will add the horizontal object detection code and fusion images after it is accepted.