Skip to content
Merged
35 changes: 26 additions & 9 deletions tutorials/cardiac-hemodynamics-assesment/notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@
{
"metadata": {},
"source": [
"# Cardiothoracic Abnormality Assessment\n",
"# Cardiothoracic Abnormality Assessment \n",
"![](images/embc_heart_tutorial_fig.png)\n",
"\n",
"In this tutorial, we demonstrate how to use low-cost, non-invasive modalities **Chest X-ray (CXR)** and **12-lead Electrocardiogram (ECG)** to assess **Cardiothoracic Abnormalities**.\n",
"\n",
"**⏱️ Estimated runtime:** Completing the steps in this tutorial will take approximately 8 minutes.\n",
"\n",
"## 📌 Problem Formulation\n",
"\n",
"We will use a multimodal dataset derived from MIMIC-CXR and MIMIC-IV-ECG, which contains approximately 50K paired CXR and ECG samples. In this tutorial, we pretrain a multimodal **CardioVAE** model using \\~49K CXR-ECG pairs via a tri-stream pretraining method. We then fine-tune this pretrained CardioVAE model on a smaller subset (\\~1K paired samples) with binary labels: **Healthy** and **Cardiothoracic Abnormality**. Lastly, we demonstrate how to interpret the trained CardioVAE model on both the CXR and ECG modalities.\n",
Expand Down Expand Up @@ -57,7 +59,10 @@
"\n",
"\n",
"**Additional Notes for Colab**\n",
"Some non-critical dependencies (e.g., `torch-geometric`) may face version conflicts when installing `pykale` on Colab. These are handled manually in the below installation step. Also an autometic crash and reset has been added as after installing this depedencies you need to restart the session. **Dont run this block again if you already did once"
"Some non-critical dependencies (e.g., `torch-geometric`) may face version conflicts when installing `pykale` on Colab. These are handled manually in the below installation step. Also an autometic crash and reset has been added as after installing this depedencies you need to restart the session. **Don't run this block again if you already did once**\n",
"\n",
"\n",
"**⏱️ Estimated runtime:** 4 minutes"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -91,7 +96,9 @@
"\n",
"As a starting point, we will mount Google Drive in Colab so that the data can be accessed directly. You might be prompted to grant permission to access your Google account—please proceed with the authorisation when asked.\n",
"\n",
"Next, we will install the required packages and load a set of helper functions to support the tutorial workflow. To keep the output clean and focused on interpretation, we also suppress unnecessary warnings."
"Next, we will install the required packages and load a set of helper functions to support the tutorial workflow. To keep the output clean and focused on interpretation, we also suppress unnecessary warnings.\n",
"\n",
"**⏱️ Estimated runtime:** 25 seconds"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -311,7 +318,9 @@
"\n",
"**Note:** \n",
"- For ease of use in Colab, the full-data loading functionality is **commented out by default**. It can be re-enabled for local or high-resource environments. \n",
"- To access the required files, ensure that the shared folder **`EMBC_workshop_data`** is added as a **shortcut to your Google Drive (under “My Drive”)**.\n"
"- To access the required files, ensure that the shared folder **`EMBC_workshop_data`** is added as a **shortcut to your Google Drive (under “My Drive”)**.\n",
"\n",
"**⏱️ Estimated runtime:** 12 seconds\n"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -366,7 +375,9 @@
"\n",
"This fine-tuning subset is explicitly selected to ensure no overlap with the samples used during pre-training, thereby simulating a realistic downstream evaluation setting.\n",
"\n",
"Unlike the fine-tuning strategy reported in *Suvon et al., MICCAI 2024*, which relied on a private in-house dataset, this approach is fully reproducible using publicly available data from MIMIC-CXR and MIMIC-IV-ECG."
"Unlike the fine-tuning strategy reported in *Suvon et al., MICCAI 2024*, which relied on a private in-house dataset, this approach is fully reproducible using publicly available data from MIMIC-CXR and MIMIC-IV-ECG.\n",
"\n",
"**⏱️ Estimated runtime:** 10 seconds"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -464,7 +475,7 @@
{
"metadata": {},
"source": [
"## Model training\n"
"## 🔥 Model training\n"
],
"cell_type": "markdown"
},
Expand All @@ -485,7 +496,9 @@
"\n",
"- Perform **joint and single-modality reconstructions**\n",
"- Optimise the **ELBO loss**, balancing image and signal modalities\n",
"- Manage logging, and reconstruction-based validation\n"
"- Manage logging, and reconstruction-based validation\n",
"\n",
"**⏱️ Estimated runtime:** 2 minutes with 1 epoch\n"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -544,7 +557,9 @@
"\n",
"- A **pretrained encoder** from `SignalImageVAE` \n",
"- A **classification head** (single or multi-layer MLP) \n",
"- A **training step** that supports standard supervised learning with cross-entropy loss\n"
"- A **training step** that supports standard supervised learning with cross-entropy loss\n",
"\n",
"**⏱️ Estimated runtime:** 1 minute with 10 epoch\n"
],
"cell_type": "markdown"
},
Expand Down Expand Up @@ -657,7 +672,9 @@
"## 🔍 Model Interpretation\n",
"\n",
"We interpret the fine-tuned model using the multimodal_signal_image_attribution interpretation method from PyKale, which builds on Captum's Integrated Gradients to generate visual explanations for both CXR and ECG inputs.\n",
"This method helps identify which regions in each modality, such as specific waveform segments in ECG and spatial regions in CXR, contributed most to the model’s prediction. This improves both transparency and clinical interpretability.\n"
"This method helps identify which regions in each modality, such as specific waveform segments in ECG and spatial regions in CXR, contributed most to the model’s prediction. This improves both transparency and clinical interpretability.\n",
"\n",
"**⏱️ Estimated runtime:** 10 seconds\n"
],
"cell_type": "markdown"
},
Expand Down