Refactor FHIR ingest to flattened YAML-driven tables

[evanfebrianto]

Summary

Adds MIMIC-IV FHIR (NDJSON) dataset support to PyHealth with a flattened, YAML-driven ingestion pipeline.

Architecture

The pipeline follows PyHealth's standard 3-step pattern:

1. Phase 1 — NDJSON → Flattened Parquet: Stream FHIR resources from *.ndjson.gz files, normalize each resource type (Patient, Encounter, Condition, Observation, MedicationRequest, Procedure) into a flat 2D Parquet table using buffered writes (50K rows/batch). Zero full-dataset-in-memory.
2. Phase 2 — Standard PyHealth pipeline: Feed the prepared Parquet tables through BaseDataset via a custom load_table override, producing global_event_df.parquet. Downstream task processing operates on standard Patient objects.

What's included

• Dataset: MIMIC4FHIRDataset — FHIR NDJSON ingest with YAML config (mimic4_fhir.yaml), streaming Parquet flattening, concept vocabulary (ConceptVocab), and CEHR timeline construction.
• Task: MPFClinicalPredictionTask — patient-level mortality prediction with MPF boundary tokens and CEHR-aligned sequences.
• Model: EHRMambaCEHR — single-stream Mamba classifier with CEHR embeddings (concept, time-delta, age, visit segment).
• Tests: Full synthetic coverage — no real MIMIC-IV data required.
• Example: End-to-end training script with quick-test mode, ablation grid, and real-data support.

Key design decisions

• base_dataset.py is unmodified — all FHIR-specific logic (UTC tz-stripping, null patient_id handling) is self-contained in MIMIC4FHIRDataset.load_table.
• Uses the existing Patient class — no custom FHIRPatient object.
• Cache identity includes schema version and YAML digest for proper invalidation.

Tested

All tests pass with synthetic data: pixi run -e test test → 905 tests OK, 67 skipped.

Contributor

Name: Evan Febrianto
NetID / email: evanf3 / evanf3@illinois.edu

Contribution type

Full pipeline (dataset + task + model) for reproducing EHRMamba-style training on tokenized FHIR timelines from MIMIC-IV FHIR NDJSON exports.

---

Note

Medium Risk
Medium risk because it replaces the MIMIC-IV FHIR ingest/storage format and task tokenization inputs, which could change cohort selection, timestamps, and downstream features despite added tests.

Overview
Refactors MIMIC4FHIRDataset ingestion from writing a single sharded global_event_df of raw FHIR JSON to streaming NDJSON normalization into flattened per-resource Parquet tables (patient, encounter, condition, observation, medication_request, procedure) under flattened_tables/, then uses a new tables: schema in mimic4_fhir.yaml plus a dataset load_table() override to rebuild global_event_df via the standard BaseDataset pipeline.

Updates cohort limiting (max_patients) to filter the normalized tables, adds glob_patterns support (and targeted PhysioNet defaults), removes the legacy FHIRPatient object path so MPF sequence building/labeling operates directly on tabular Patient rows, and refreshes the example script/docs/tests accordingly (including deceased-boolean parsing safeguards and new flattened-column assertions).

^{Reviewed by Cursor Bugbot for commit a5b6aac. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a5b6aac. Configure here.}

[cursor]

Encounter duplicate IDs silently drop earlier visit index

Low Severity

encounter_visit_idx and encounter_start_by_id are built from encounter_rows via dict comprehensions that iterate in sorted-time order. If two encounter rows share the same encounter_id (possible in messy FHIR exports or after deduplication failures), the later entry silently overwrites the earlier one. Events linked to that ID would receive the wrong visit_idx and start time, corrupting CEHR sequence visit segmentation without any warning.

Additional Locations (1)

• pyhealth/datasets/mimic4_fhir.py#L797-L800

 

^{Reviewed by Cursor Bugbot for commit a5b6aac. Configure here.}