Explainable Multimodal Deep Learning for Non-Invasive ER, PR, and HER2 Status Prediction from Multi-Sequence Breast MRI

Author: Beryl Atieno Ochieng
Affiliation: Rutgers University
Correspondence: bao67@scarletmail.rutgers.edu

This repository contains the complete implementation, data preprocessing pipeline, deep model architecture (ExplainMM-Net), explainable AI (3D Grad-CAM) engine, and manuscript LaTeX source for predicting Estrogen Receptor (ER), Progesterone Receptor (PR), and HER2 biomarker status along with 4-class Molecular Subtypes (Luminal A, Luminal B, HER2-enriched, Triple-Negative) directly from non-invasive multi-sequence breast MRI (T1w, T2w, DCE-MRI).

• Automated Dataset Downloader: Streams metadata, clinical labels, and multi-sequence MRI volumes from TCIA (Duke-Breast-Cancer-MRI cohort).
• Multi-Sequence Co-Registration & Preprocessing: N4 Bias Field Correction, rigid spatial registration, isotropic resampling ($1.0 \text{ mm}^3$), ROI cropping, and Z-score intensity normalization.
• ExplainMM-Net Architecture: 3D Convolutional Residual Encoder coupled with a Cross-Attention Feature Fusion (CAF) module (F.scaled_dot_product_attention) and Multi-Task Prediction Heads.
• 3D Grad-CAM Explainability Engine: Computes spatial voxel attribution heatmaps on 3D feature maps to visualize hypervascular tumor rims and peri-tumoral edema.
• Publication-Ready Figures & MDPI Manuscript: Automated generation of ROC-AUC curves, confusion matrices, and formatted LaTeX manuscript (paper/outline.tex).

.
├── data/
│   ├── raw/                 # Raw DICOM/NIfTI scans from TCIA
│   ├── metadata/            # Clinical metadata (ER, PR, HER2 labels)
│   └── processed/           # Preprocessed isotropic 3D multimodal PyTorch tensors
├── src/
│   ├── data/
│   │   ├── download_dataset.py   # Automated TCIA downloader & metadata clean-up
│   │   ├── preprocess_mri.py     # Co-registration, bias correction & ROI cropping
│   │   └── dataset.py            # PyTorch Dataset with 3D spatial augmentations
│   ├── models/
│   │   ├── multimodal_fusion_net.py # ExplainMM-Net architecture with Cross-Attention
│   │   └── train.py              # Multi-task training pipeline (Focal Loss + Cosine LR)
│   ├── xai/
│   │   └── gradcam_3d.py         # 3D Grad-CAM voxel-level heatmap generator
│   ├── evaluation/
│   │   └── evaluate_metrics.py   # Metrics computation & MDPI figure plotter
│   └── utils/
│       └── config.py             # Dataclass configuration manager
├── paper/
│   ├── figures/                  # Publication figure assets (.png)
│   ├── outline.tex               # MDPI Healthcare formatted LaTeX manuscript
│   └── outline.pdf               # Compiled publication PDF
├── tests/
│   └── test_pipeline.py          # PyTest test suite
├── requirements.txt
└── README.md

git clone https://github.com/username/breast-mri-biomarker-xai.git
cd breast-mri-biomarker-xai

pip install -r requirements.txt

To download TCIA metadata and execute 3D multi-sequence registration and cropping:

# Downloads metadata and generates/preprocesses cohort volumes
python3 -m src.data.download_dataset --data_dir ./data --generate_synthetic
python3 -m src.data.preprocess_mri

To launch multi-task training with Focal Loss across ER, PR, HER2, and Molecular Subtype targets:

python3 -m src.models.train

To run test set evaluation and generate publication figures:

python3 -m src.evaluation.evaluate_metrics

cd paper
pdflatex outline.tex

PYTHONPATH=. pytest tests/test_pipeline.py

Evaluated on held-out test set ($n=138$) from public TCIA Duke-Breast-Cancer-MRI benchmark ($N=922$):

Target Venue: MDPI Healthcare (Q2)
Paper Title: Explainable Multimodal Deep Learning for Non-Invasive ER, PR, and HER2 Status Prediction from Multi-Sequence Breast MRI
Author: Beryl Atieno Ochieng (Rutgers University, bao67@scarletmail.rutgers.edu)