MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Models

Open the pipeline PDF

MedLoCoMo generates long-context, multi-session medical dialogue benchmark samples from MIMIC-IV v3.1 and MIMIC-IV-Note v2.2.

• One patient = one MedLoCoMo sample
• One admission = one session
• Only two speakers: Doctor and Patient
• Dialogues are grounded in:
  • discharge notes
  • radiology notes
  • diagnoses ICD
  • procedures ICD
  • microbiology

Admissions without discharge notes are skipped. Re-running the same patient replaces that patient's output directory instead of appending to it.

The core benchmark artifacts for each patient are combined_conversation.json, the full chronological multi-session dialogue, and benchmark_qa.json, the final QA set paired with that dialogue.

The released benchmark data is included in this repository under MedLoCoMo/.

Install

From the repository root:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Dataset Layout

Expected defaults:

data/
  mimic-iv/
    hosp/
  mimic-iv-notes/

Commands

Generate the MedLoCoMo dialogue artifacts for one patient:

export OPENAI_API_KEY="YOUR_KEY"
python main.py generate-patient --subject-id 10000032 --output-root MedLoCoMo

Override dataset, model, or output paths:

python main.py generate-patient \
  --mimiciv-dir /path/to/mimic-iv \
  --mimiciv-note-dir /path/to/mimic-iv-notes \
  --output-root MedLoCoMo \
  --subject-id 10000032 \
  --model gpt-5.2

Generate QA for one patient after the dialogue artifacts exist:

python main.py generate-qa \
  --subject-id 10000032 \
  --output-root MedLoCoMo \
  --model gpt-5.2

Generate dialogue and QA artifacts together:

python main.py generate-all \
  --subject-id 10000032 \
  --output-root MedLoCoMo \
  --model gpt-5.2

generate-qa requires that every admission folder already contains both conversation.json and summary.json.

MedLoCoMo Benchmark Layout

MedLoCoMo/
  <subject_id>/
    combined_conversation.json  # core chronological multi-session dialogue
    patient_summary.json
    benchmark_qa.json           # core benchmark QA set
    cross_admission_qa.json
    <hadm_id>/
      formed_packet.json
      prompt_record.json
      conversation.json
      summary.json
      qa.json

Per-patient generation writes to MedLoCoMo/_tmp/<subject_id>/ first and only replaces MedLoCoMo/<subject_id>/ after a successful run.

QA generation writes to MedLoCoMo/_tmp/qa_<subject_id>/ first and only replaces QA artifacts on success.