Skip to content

fixes#25

Merged
will-pang merged 5 commits into
mainfrom
wp/add-xray-to-multimodal-task
May 28, 2026
Merged

fixes#25
will-pang merged 5 commits into
mainfrom
wp/add-xray-to-multimodal-task

Conversation

@jhnwu3

@jhnwu3 jhnwu3 commented Apr 16, 2026

Copy link
Copy Markdown

This pull request adds new multimodal embedding runners for MIMIC-IV + CXR using both MLP and JambaEHR architectures, and brings several improvements and standardizations to the Mamba-based runner. The changes improve reproducibility, flexibility, and evaluation reporting for multimodal mortality prediction experiments.

New multimodal runner scripts:

  • Added a new script multimodal_embedding_mlp_mimic4_cxr.py for running unified multimodal embeddings with an MLP backbone. This script supports EHR, notes, and X-ray metadata/negbio inputs, and includes argument parsing, dataset setup, model instantiation, training, evaluation, and benchmarking.
  • Added a new script multimodal_embedding_jamba_mimic4_cxr.py for running unified multimodal embeddings with a JambaEHR (hybrid Transformer + Mamba) backbone. The script mirrors the structure and features of the MLP runner, supporting flexible configuration and benchmarking.

Enhancements and standardization to the Mamba runner:

Evaluation and metrics improvements:

  • Changed the evaluation metrics in the Mamba runner (multimodal_embedding_mamba_mimic4_cxr.py) from only "accuracy" to include "pr_auc" and "roc_auc", aligning with the new runner scripts for better model assessment.
  • Added code to print evaluation metrics after inference, providing clearer benchmarking output. [1] [2]

Argument parsing and reproducibility:

  • Added a --condor flag to all runner scripts to facilitate setting the device to "cuda" (no index) for HTCondor GPU jobs. [1] [2] [3]
  • Standardized the default batch size to 1 in the Mamba runner for consistency with the new scripts.

These updates provide a unified, extensible framework for benchmarking multimodal models on MIMIC-IV with CXR data, and facilitate reproducible experiments across different neural architectures.

References:

@will-pang will-pang self-requested a review May 28, 2026 18:47

@will-pang will-pang left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't test extensively, but I trust John haha

@will-pang will-pang merged commit 6a96ed8 into main May 28, 2026
@will-pang will-pang deleted the wp/add-xray-to-multimodal-task branch June 3, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants