CSF-net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

Proposed method

This code is a pytorch implementation of our paper "CSF-net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting". It consists of three components: (a) spatial feature extraction module, (b) temporal residual fusion module, and (c) CMAF module.The spatial module combines ResNet with the CBAM for enhanced feature representation. The temporal residual fusion module integrates features from different time points to capture correlations. The CMAF module uses cross-modal attention to effectively integrate follow-up and clinical data.The figure below shows our proposed network.

The Temporal Residual Fusion module in our method

Experiment result

We compared our results with other state-of-the-art methods, and our results were better than any other comparison method. The results indicate that the introduced CMAF module can effectively capture the intrinsic correlation between the two modalities. The time residual module we designed has good sensitivity to features at different times, and these two modules significantly improve the performance of the model.

Methods	Acc	Prec	F1 score	AUC	Rec
SCANs	0.7865	0.7667	0.7077	0.7725	0.6571
NAS-Lung	0.8539	0.8235	0.8116	0.8910	0.8000
T-LSTM	0.7645	0.7012	0.6527	0.7778	0.6000
DeepCAD	0.8590	0.7879	0.8254	0.8990	0.8667
MFCN	0.7949	0.7059	0.7500	0.8903	0.8000
RadFusion	0.7753	0.8026	0.6667	0.7693	0.6000
CSF-Net (ours)	0.8974	0.8235	0.8750	0.9389	0.9333

Pre-requisties

• Linux

• Python>=3.7

• NVIDIA GPU + CUDA12.1 cuDNN8.9

Getting started to evaluate

Install dependencies

pip install -r requirements.txt

Data preprocess

For the CT images, we utilized the preprocessing method from the code available at https://github.com/lfz/DSB2017, and created ROI files based on the nodule coordinates provided by doctors. For the clinical information, we applied word embeddings and performed dimensionality reduction to fit the model's input.

Evaluation

To do the evaluation process, please run the following command :

python main.py

Data

We used the National Lung Screening Trial (NLST) dataset, with the original data available for download at https://cdas.cancer.gov/learn/nlst/images/. From this dataset, we selected 443 cases based on pathological gold standards, which we named NLST-cmst.