Reproducibility gap in ICU pathogen prediction benchmarks (MIMIC-IV)

[netanelcyber]

Hi PyHealth team,

I have been working on an open-source ICU pathogen prediction pipeline using MIMIC-IV:
https://github.com/netanelcyber/PenuX

During development we noticed something that may be relevant for the broader clinical ML community:

Observation

Many ICU prediction pipelines report strong AUROC values, but relatively few evaluate:

• temporal drift
• calibration stability
• subgroup robustness
• rare-class confidence behavior
• leakage sensitivity across ICU workflows

In our experiments, relatively small preprocessing choices produced unexpectedly large differences in:

• pathogen ranking stability
• calibration curves
• minority-class behavior
• external-like temporal splits

This raises a broader reproducibility question:

Are current clinical ML benchmark pipelines sufficiently robust to temporal and operational shifts commonly seen in ICU environments?

Potential contribution ideas

I would be interested in contributing PyHealth-compatible examples for:

• reproducible MIMIC-IV infection/pathogen prediction
• calibration-first evaluation
• temporal split utilities
• subgroup robustness analysis
• uncertainty estimation benchmarks
• eICU transfer experiments
• clinically interpretable evaluation templates

Technical direction

Current experiments include combinations of:

• tabular clinical variables
• vitals/labs trajectories
• ICU time-series aggregation
• calibration analysis
• ranking-oriented evaluation rather than pure hard classification

Questions for maintainers/community

1. Are there existing efforts around temporal robustness benchmarking in PyHealth?
2. Would calibration-focused benchmark examples be useful for the project?
3. Is there interest in a community benchmark around ICU robustness / reproducibility failures?

I would also appreciate feedback from others working on:

• MIMIC-IV
• eICU
• clinical foundation models
• ICU time-series
• trustworthy medical AI

Project:
https://github.com/netanelcyber/PenuX

Thanks again for building PyHealth — it has been extremely useful for rapid experimentation in clinical ML.

[jhnwu3]

1. Short answer is yes.
2. Also yes, we have pyhealth.calib in case you're interested and a research paper recently accepted investigating CP on healthcare tasks.
3. Yeah, for sure.

These are all very aligned with our interests. Happy to chat more on discord. We're trying to gather more like minded people, and make PyHealth more accessible for the wider community, so let us know what we can do to help.

One bottleneck with all benchmarks imo in healthcare is they're not flexible enough for adaptation to other people's workflows, and tbh quite frankly they take too long to run. I think if you take your benchmark into the Pyhealth format, you'll be able to functionally make it more accessible to others with our backend. If you look at our benchmark numbers in our recently updated paper, a 3 year old MacBook pro is funnily enough faster than our 20k usd workstation for data processing.