PE-MVCNET: MULTI-VIEW AND CROSS-MODAL FUSION NETWORK FOR PULMONARY EMBOLISM PREDICTION

Our paper has been accepted by ISBI 2024!

Proposed method

We spend a lot of time collecting and summarizing relevant papers and datasets, where you can find them at https://github.com/Ivygugu/Pulmonary-Embolism-Detection/blob/main/README.md

This code is a pytorch implementation of our paper "PE-MVCNET: MULTI-VIEW AND CROSS-MODAL FUSION NETWORK FOR PULMONARY EMBOLISM PREDICTION".

Our method comprises the Image-only module with an integrated multi-view block, the EMR-only module, and the Cross-modal Attention Fusion (CMAF) module. These modules cooperate to extract comprehensive features that subsequently generate predictions for PE. The figure below shows our proposed network.

The Multi-View Coupled Self-Attention Block of our method

Experiment result

We compared to other state-of-the-art methods and our results are better than any other method of comparison. Results prove that the introduced CMAF module adeptly captures the inherent correlations between the two modalities, thereby providing the model with richer information.

Methods	AUROC	ACC	F1 score	Specificity	Sensitivity	PPV	NPV
3D ResNet50	0.694	0.556	0.687	0.785	0.963	0.534	0.785
3D ResNet101	0.722	0.611	0.701	0.757	0.902	0.574	0.757
PENet	0.660	0.623	0.666	0.656	0.743	0.604	0.656
PEfusion	0.936	0.882	0.882	0.900	0.866	0.898	0.867
PE-MVCNet(Ours)	0.941	0.902	0.906	0.932	0.939	0.899	0.932

Pre-requisties

• Linux

• Python>=3.7

• NVIDIA GPU (memory>=23G) + CUDA cuDNN

Getting started to evaluate

Install dependencies

pip install -r requirements.txt

Download the checkpoint

Our model's best checkpoint is provided in this link https://drive.google.com/file/d/1RAhGWlnhfWrR4VFcd0w_a_pskoMam7hd/view?usp=drive_link, and you can use it to easily replicate our results.

Data preprocess

The preprocessing of the image data can follow this link https://github.com/marshuang80/PENet In short, using create_hdf5.py to make an hdf5 file. As for the preprocessing of the tabular data, We first conduct dimensionality reduction using LinearSVC on the EMR data, then use TabNet to transform the data into suitable embeddings.

Evaluation

To do the evaluation process, please run the following command :

sh test.sh

If you want to find the best threshold for the result, please run the following command :

python best_threshold.py

Train by yourself

If you want to train by yourself, you can run this command :

sh train.sh

Data

The data we used is from Stanford University Medical Center dataset. You can download it from https://stanfordaimi.azurewebsites.net/datasets/3a7548a4-8f65-4ab7-85fa-3d68c9efc1bd.

And the tabular data is already provided in this link.