IMF

Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration [IEEE TCSVT2024]

By Di Wang, Jinyuan Liu, Long Ma, Risheng Liu, and Xin Fan*

Updates

[2024-06-08] Our paper is accepted to IEEE TCSVT !

[2023-08-25] Our paper is available online! [arXiv version]

Requirements

• CUDA 10.1
• Python 3.6 (or later)
• Pytorch 1.6.0
• Torchvision 0.7.0
• OpenCV 3.4
• Kornia 0.5.11

Data preparation

1. You can generate misaligned infrared-visible images for training/testing by

      cd ./data
      python generate_affine_deform_data.py

In 'Trainer/train_reg.py', deformable infrared images are generated in real time by default during training.

2. You can obtain self-visual saliency maps for training the fusion process of infrared and visible images by

      cd ./data
      python get_svs_map_softmax.py

Get start

1. You can use the pseudo infrared images [link code: qqyj] generated by the CPSTN proposed by UMF to train/test our C-MPDR:

      cd ./Trainer
      python train_reg.py
   
      cd ./Test
      python test_reg.py

2. If you tend to train Registration and Fusion subnetworks separately, You can run following commands:

      cd ./Trainer
      python train_reg.py
   
      cd ./Trainer
      python train_co_fuse.py

The corresponding test code 'test_reg.py' and 'test_co_fuse.py' can be found in 'Test' folder.

3. If you tend to train Registration and Fusion subnetworks jointly, You can run following command:

       cd ./Trainer
       python train_reg_co_fusion_sa.py

The corresponding test code 'test_reg_co_fusion.py' can be found in 'Test' folder.

Dataset

Please download the following datasets:

• RoadScene
• TNO
• M3FD
• MSIFT

Note: The above datasets are manually pre-registered. Desired misaligned image can be generated using the proposed image synthesis method.

Pretrained Models

1. Pretrained models of registration subnetwork MPDR are as follows:

• RoadScene (code: wyi0)
• TNO (code: qul9)
• M3FD (code: uq0b)
• MSIFT (code: 410b)

2. Pretrained models of fusion subnetwork TCF are as follows:

• RoadScene (code: nju6)
• TNO (code: rqmx)
• M3FD (code: 0rd4)

Experimental Results

1. Please download the Registration results by our IMFusion :

• RoadScene (code: pv7s)
• TNO (code: yyf8)
• M3FD (code: 7812)
• MSIFT (code: ahfs)

2. Please download the Fusion results by our IMFusion :

• RoadScene (code: 5rbw)
• TNO (code: y29p)
• M3FD (code: 2x64)

Citation

@article{Wang_2023_IMF,
      author={Di Wang and Jinyuan Liu and Long Ma and Risheng Liu and Xin Fan},
      title={Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration}, 
      journal={{IEEE} Transactions on Circuits and Systems for Video Technology},
      year={2024}
}