MOSCARD - Multimodal Opportunistic Screening for Cardiovascular Adverse events with Causal Reasoning and De-confounding

Our study addresses bias in multimodal medical imaging by integrating causal reasoning techniques. We utilize chest X-ray (CXR) images as the primary source of information and employ electrocardiogram (ECG) signals as a complementary guiding modality. To effectively preserve and leverage the essential features from CXR images while incorporating insights from ECG data, we have adapted a co-attention mechanism originally developed for processing H&E stained whole slide images alongside genomic factors. For single modality training, we employ a Vision Transformer (ViT) architecture, specifically utilizing the MedCLIP image modality, to serve as a unified encoder for both ECG signals and CXR images during the encoder training phase. This integration allows for a cohesive and comprehensive analysis of the multimodal medical data.

Model framework and Causal graph

• Model overview:

Proposed MOSCARD architecture and de-confounding causal reasoning graph, input X, task label Y, causal factor A, confounder C, directed edges for causal confounder relations: (a) Step 1 – single modality encoder training with confusion loss; (b) Step 2 multimodal learning with co-attention and SCM; (c) Step 1 training – Single modality; (d) Step 2 Multimodal training with co-attention and causal intervention.

Performance: Comparative model performance after multimodal data alignment with combined and individual modality (95% confidence intervals using bootstrapping).

Baseline Multimodal
Dataset Types	MACE	CXR+ECG (Combined)		CXR+ECG (CXR)		CXR+ECG (ECG)
		Accuracy	AUC	Accuracy	AUC	Accuracy	AUC
Internal datasets (PCI)	MACE_6M	0.653[0.652, 0.655]	0.711[0.709, 0.712]	0.643[0.642, 0.645]	0.690[0.688, 0.690]	0.639[0.637, 0.641]	0.681[0.678, 0.682]
	MACE_1yr	0.663[0.663, 0.665]	0.725[0.723, 0.726]	0.651[0.649, 0.653]	0.705[0.703, 0.708]	0.646[0.645, 0.649]	0.688[0.685, 0.691]
	MACE_2yr	0.656[0.653, 0.658]	0.717[0.716, 0.719]	0.647[0.645, 0.648]	0.702[0.698, 0.704]	0.646[0.646, 0.649]	0.690[0.687, 0.691]
	MACE_5yr	0.653[0.651, 0.655]	0.712[0.710, 0.715]	0.640[0.635, 0.640]	0.698[0.696, 0.699]	0.642[0.641, 0.644]	0.687[0.684, 0.690]
External datasets (MIMIC)	MACE_6M	0.634[0.614, 0.640]	0.662[0.654, 0.679]	0.606[0.590, 0.616]	0.630[0.623, 0.653]	0.634[0.617, 0.640]	0.711[0.692, 0.724]
External datasets (ED)	MACE_1yr	0.715[0.708, 0.715]	0.792[0.789, 0.796]	0.739[0.729, 0.737]	0.804[0.800, 0.809]	0.672[0.668, 0.678]	0.737[0.733, 0.742]
Causal Multimodal
Dataset Types	MACE	CXR+ECG (Combined)		CXR+ECG (CXR)		CXR+ECG (ECG)
		Accuracy	AUC	Accuracy	AUC	Accuracy	AUC
Internal datasets (PCI)	MACE_6M	0.652[0.649, 0.653]	0.711[0.707, 0.711]	0.644[0.643, 0.647]	0.695[0.692, 0.696]	0.641[0.640, 0.645]	0.686[0.684, 0.687]
	MACE_1yr	0.662[0.661, 0.663]	0.724[0.723, 0.726]	0.656[0.653, 0.658]	0.713[0.711, 0.714]	0.647[0.645, 0.648]	0.694[0.693, 0.696]
	MACE_2yr	0.654[0.653, 0.656]	0.715[0.714, 0.718]	0.648[0.646, 0.649]	0.706[0.704, 0.708]	0.649[0.649, 0.652]	0.695[0.695, 0.698]
	MACE_5yr	0.650[0.647, 0.651]	0.711[0.708, 0.711]	0.647[0.645, 0.649]	0.703[0.701, 0.704]	0.642[0.640, 0.644]	0.687[0.685, 0.688]
External datasets (MIMIC)	MACE_6M	0.669[0.658, 0.677]	0.677[0.662, 0.690]	0.623[0.619, 0.642]	0.638[0.620, 0.649]	0.640[0.627, 0.650]	0.710[0.685, 0.713]
External datasets (ED)	MACE_1yr	0.737[0.734, 0.741]	0.810[0.806, 0.813]	0.772[0.767, 0.773]	0.837[0.833, 0.839]	0.673[0.671, 0.680]	0.753[0.748, 0.761]
Conf Multimodal
Dataset Types	MACE	CXR+ECG (Combined)		CXR+ECG (CXR)		CXR+ECG (ECG)
		Accuracy	AUC	Accuracy	AUC	Accuracy	AUC
Internal datasets (PCI)	MACE_6M	0.681[0.681, 0.684]	0.737[0.736, 0.739]	0.671[0.671, 0.673]	0.722[0.720, 0.724]	0.666[0.665, 0.668]	0.721[0.719, 0.722]
	MACE_1yr	0.691[0.690, 0.693]	0.751[0.750, 0.752]	0.678[0.675, 0.678]	0.737[0.736, 0.741]	0.671[0.669, 0.672]	0.730[0.728, 0.731]
	MACE_2yr	0.683[0.680, 0.687]	0.745[0.743, 0.746]	0.672[0.671, 0.674]	0.730[0.730, 0.735]	0.673[0.672, 0.675]	0.732[0.731, 0.736]
	MACE_5yr	0.679[0.678, 0.681]	0.740[0.739, 0.742]	0.664[0.661, 0.665]	0.723[0.721, 0.725]	0.672[0.672, 0.674]	0.727[0.724, 0.727]
External datasets (MIMIC)	MACE_6M	0.623[0.619, 0.641]	0.673[0.658, 0.683]	0.594[0.587, 0.609]	0.649[0.634, 0.668]	0.663[0.647, 0.670]	0.649[0.632, 0.667]
External datasets (ED)	MACE_1yr	0.701[0.697, 0.707]	0.777[0.773, 0.782]	0.728[0.722, 0.728]	0.789[0.779, 0.787]	0.635[0.635, 0.643]	0.693[0.687, 0.696]
CaConf Multimodal
Dataset Types	MACE	CXR+ECG (Combined)		CXR+ECG (CXR)		CXR+ECG (ECG)
		Accuracy	AUC	Accuracy	AUC	Accuracy	AUC
Internal datasets (PCI)	MACE_6M	0.681[0.679, 0.682]	0.733[0.732, 0.735]	0.670[0.668, 0.670]	0.716[0.715, 0.718]	0.673[0.671, 0.674]	0.723[0.722, 0.725]
	MACE_1yr	0.691[0.689, 0.692]	0.750[0.749, 0.752]	0.681[0.676, 0.683]	0.734[0.733, 0.736]	0.674[0.672, 0.676]	0.735[0.734, 0.738]
	MACE_2yr	0.683[0.681, 0.684]	0.740[0.738, 0.742]	0.669[0.665, 0.670]	0.723[0.723, 0.726]	0.675[0.673, 0.676]	0.733[0.731, 0.734]
	MACE_5yr	0.678[0.677, 0.679]	0.735[0.733, 0.738]	0.669[0.667, 0.670]	0.717[0.716, 0.719]	0.675[0.673, 0.676]	0.730[0.729, 0.735]
External datasets (MIMIC)	MACE_6M	0.623[0.614, 0.635]	0.662[0.649, 0.676]	0.600[0.590, 0.615]	0.644[0.624, 0.658]	0.611[0.594, 0.619]	0.657[0.633, 0.670]
External datasets (ED)	MACE_1yr	0.702[0.697, 0.704]	0.777[0.768, 0.783]	0.732[0.727, 0.734]	0.809[0.799, 0.811]	0.647[0.645, 0.653]	0.710[0.705, 0.714]

Train the models

• Data preparation

  • Prepare your data by following the example provided in config/train.csv.
  • Update the data path in config/config.json.
  • Convert ECG signals into image representations using the code example in ECG/data/ecg_convert_example.ipynb (reference from ecg_plot library)
  • Delete all the lateral images of chest X-ray datasets by running the example command: python CXR/view_clf/inference.py
  • Convert chest X-ray DICOM to PNG (reference from (https://github.com/ramon349/DicomProcTools/blob/main/A3IDicomTools/extractors/PngExtractor.py))

• Model Training Ensure all necessary packages are installed by running: pip install -r requirements.txt

  • Single modality training

    • CXR single modality Baseline follow the on-screen prompt to choose a mode (1: Baseline single modality encoder training; 2: Single modality encoder training with confusion loss): bash CXR/scrpits/train.sh
    • ECG single modality Baseline follow the on-screen prompt to choose a mode (1: Baseline single modality encoder training; 2: Single modality encoder training with confusion loss): bash ECG/scrpits/train.sh

  • MOSCARD training

    • Train follow the on-screen prompt to choose a mode (1–4): bash MOSCARD/scrpits/train.sh

    Option	Mode Name	Description
    1	Baseline	Trains a baseline multimodal model using pre-trained ECG and CXR backbones without de-confounding and causal reasoning.
    2	Causal	Trains the model with causal reasoning mechanisms based on Baseline.
    3	Conf	Trains the single baseline model using backbones that were trained with de-confounding strategies.
    4	CaConf	Trains the causal model using de-confounding backbones (Final Proposed model).

  • MedCLIP Baseline training: Train follow the on-screen prompt to choose a mode (1-2): bash MedClip_baseline/scripts/train.sh

    • Mode 1 focuses on learning shared representations between modalities through CLIP-based alignment and cross-attention.
    • Mode 2 performs downstream classification by freezing the alignment backbone and training only the final MLP classifiers.

  • ALBEF Baseline training reference is from code.

• Model Testing

  • Single modality testing
    • CXR single modality: bash CXR/scrpits/train.sh
    • ECG single modality: bash ECG/scrpits/train.sh
  • MOSCARD training testing: bash MOSCARD/scrpits/test.sh
  • MedCLIP Baseline testing: bash MedClip_baseline/scripts/test.sh
  • ALBEF Baseline testing reference is from code.

• Model Weights

  • Model weights for the proposed MOSCARD model, including Step 1: single-modality training weights, and Step 2: multi-modal classification training weights. Location: google drive

• Saliency map figure To plot a saliency map, you can refer to the code.

Contact

• If you have any quesions, please post it on github issues or email me.

Reference

• https://github.com/mahmoodlab/MCAT/tree/master?tab=readme-ov-file
• https://github.com/RyanWangZf/MedCLIP
• https://github.com/dy1901/ecg_plot/tree/master
• https://github.com/adityac94/Grad_CAM_plus_plus/tree/master