Liying Wang, Xiaoli Zhang*, Chuanmin Jia*, and Siwei Ma
- [2024/12] The manuscript has been submitted to IEEE Transactions on Image Processing.
- [2025/9] We have received the acceptance notification😊
Infrared-visible image fusion methods aim at generating fused images with good visual quality and also facilitate the performance of high-level tasks. Indeed, existing semantic-driven methods have considered semantic information injection for downstream applications. However, none of them investigates the potential for reciprocal promotion between pixel-wise image fusion and cross-modal feature fusion perception tasks from a macroscopic task-level perspective. To address this limitation, we propose a unified network for image fusion and semantic segmentation. MAFS is a parallel structure, containing a fusion sub-network and a segmentation sub-network. On the one hand, we devise a heterogeneous feature fusion strategy to enhance semantic-aware capabilities for image fusion. On the other hand, by cascading the fusion sub-network and a segmentation backbone, segmentation-related knowledge is transferred to promote feature-level fusion-based segmentation. Within the framework, we design a novel multi-stage Transformer decoder to aggregate fine-grained multi-scale fused features efficiently. Additionally, a dynamic factor based on the max-min fairness allocation principle is introduced to generate adaptive weights of two tasks and guarantee smooth training in a multi-task manner.
Please run fusion.py and segmentation_test.py.
And the training part will be open soon.
1. Virtual Environment
# create virtual environment
conda create -n MAFS python=3.8.0
conda activate MAFS
# select the pytorch-gpu version yourself
# install MAFS requirements
pip install -r requirements.txt
Please note: the provided requirements.txt was generated from our environment and may depend on specific hardware/software settings. Ensure compatibility with your own machine before installation😊
2. Training Dataset
The Training and test datasets are strictly consistent with the original dataset split. For more details, see the relevant repositories: MFNet, PST900, and FMB. Besides, MAFS's checkpoints can be downloaded from this link. Code: mafs. The additional choice is Google Drive.
@article{wang2025mafs,
title={MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation},
author={Wang, Liying and Zhang, Xiaoli and Jia, Chuanmin and Ma, Siwei},
journal={IEEE Transactions on Image Processing},
year={2025},
publisher={IEEE}
}
If you have any corresponding questions, please feel free to contact Liying Wang at my_lnnu@163.com or liyingw23@mails.jlu.edu.cn.
Our code is built based on PSFusion and DeFusion. SegMiF and CDDFuse primarily inspire this work. Please also follow their licenses. If you are inspired by MRFS, please also cite their paper. We sincerely thank Xihang Hu's valuable suggestion during the experimental implementation. If you're interested in the field of salient object detection or camouflaged object detection, please refer to hu-xh. Thank you all for your attention.