MedHyperGraph

EHR-Integrated Multimodal Hyperedges for Clinical VQA MICCAI 2026 Workshop MSB EMERGE · OpenReview · Demo video

A chest X-ray is rarely interpreted in isolation. Whether a finding is new, unchanged, expected after a procedure, or consistent with a known diagnosis is usually not visible in a single image — it depends on the patient's prior studies, earlier reports, diagnoses, and the timing between events. Most medical vision-language models still answer isolated image–question pairs.

MedHyperGraph builds a per-patient multimodal hypergraph over structured EHR events, prior-report entities, imaging studies, ontology concepts and image regions; retrieves a small evidence subgraph under explicit temporal constraints; and passes it, with the current chest X-ray, to a VLM. No task-specific training.

appendix.md is the implementation companion to this document: how the hypergraph is built stage by stage, the retrieval configuration, the verbatim prompts, the evidence-linearisation format, and the run settings behind each reported table.

Demo

The evidence viewer, showing which parts of a patient's record the retriever actually used to answer a question — see Web UI to run it yourself.

Results

Boolean accuracy / balanced accuracy on the HyperWalker-filtered EHRXQA test set (n = 1900). The answers are imbalanced (56 % "no") so a constant "no" scores 56.0 / 50.0, and balanced accuracy is the primary metric.

Method	MedGemma 1.5 4B	MedGemma 1 27B	Qwen 3.6-35B
Vanilla	59.26 / 54.99	58.11 / 56.54	60.00 / 57.67
text-RAG	58.47 / 54.84	58.94 / 58.13	67.89 / 66.66
GraphRAG	58.74 / 54.88	58.84 / 54.92	64.84 / 62.40
MedHyperGraph	64.63 / 63.61	69.40 / 69.13	73.95 / 73.64

---

How we ran it

The setup the reported numbers were produced on.

Python	3.12+
GPUs	2 × 24 GB, tensor-parallel 2 — the Qwen AWQ weights are ~25 GB and will not fit a single card
Inference server	vLLM, OpenAI-compatible, on :8080
Data	PhysioNet credentialled access — see below

Install

python -m venv .venv && source .venv/bin/activate
pip install -e .
cp configs/default.example.yaml configs/default.yaml   # then edit the paths
medhg doctor                                           # validates path

Data

Everything this project reads is PhysioNet credentialled-access and is not distributed here. You need your own credentials and data use agreements. This repository ships code only: result artifacts embed verbatim radiology reports, so they cannot be redistributed either.

Dataset	Used for
MIMIC-CXR-JPG 2.1.0	chest X-ray images attached to the answer call
MIMIC-CXR	free-text radiology reports, the source of report entities
MIMIC-IV	structured EHR: admissions, diagnoses, procedures, prescriptions, labs, transfers
EHRXQA	the question set and the tb_cxr study↔patient mapping

Each requires completing the CITI training and signing the data use agreement. None may be redistributed.

Model	Used for
Qwen3.6-35B-A3B-AWQ	the backbone that produced the headline result
MedGemma 1.5 4B-IT / MedGemma 27B-IT	the two MedGemma backbones
BioViL-T (hi-ml-multimodal)	optional, only for --grounding. Every reported result ran without it.

Directory layout

The paths in configs/default.yaml expect MIMIC's two-level prefix layout:

mimic-cxr-jpg/.../files/p10/p10000032/s90000001/<image_id>.jpg
MIMIC-Reports/files/p10/p10000032/s90000001.txt
ehrxqa/.../database/gold/tb_cxr.csv
ehrxqa/.../database/silver/tb_cxr.csv
EHRXQA_per_Patient/<subject_id>/...

Both EHRXQA splits are needed. paths.ehrxqa_db points at gold/, and the code also reads the silver/ sibling. The cohort spans both (393 patients in gold, 778 in silver) and they are disjoint (0 study_id and 0 subject_id overlap). Reading only one makes the other's patients fail with "No studies found".

A note on the dates

EHXQA timestamps are shifted into the future during de-identification, so years such as 2105 are expected rather than errors. The prompt anchor date is 2105-12-31, and every relative temporal expression in a question resolves against it.

Serve the Qwen backbone

One model at a time, on port 8080.

# our run settings
CUDA_VISIBLE_DEVICES=0,1 vllm serve $MODELS/Qwen3.6-35B-A3B-AWQ \
  --served-model-name Qwen3.6-AWQ --tensor-parallel-size 2 --port 8080 \
  --max-model-len 98304 --reasoning-parser qwen3 \
   --max-num-seqs 64 \
  --gpu-memory-utilization 0.92 --mm-processor-cache-gb 0

configs/backbones/qwen35.yaml carries the matching client settings: top_k 5, max_token_for_text_unit 250, max_tokens 16384, temperature 0.6, repetition_penalty 1.0, workers 16.

Build the graphs

medhg doctor                                            # check the paths first
python -m medhypergraph.graph.ingest_reports --config configs/default.yaml --patient <pid>
python -m medhypergraph.graph.ingest_ehr     --config configs/default.yaml --patient <pid>
python -m medhypergraph.graph.merge_ehr      --config configs/default.yaml --patients <pid>

Expect a full-cohort build to take a long time: extraction is one VLM call per report segment across 870 patients. merge_ehr needs paths.ehr_source_dir, a second tree of per-patient graphs carrying dated EHR chunks, built separately; without it a rebuild is missing every uppercase-typed EHR entity and the published graphs cannot be reproduced exactly. A rebuild was measured against the published graph at −0.5 % nodes, −0.5 % edges, node Jaccard 0.988.

Rebuilding on a thinking model requires thinking off for extraction. With it on, vLLM returns the response in reasoning and leaves content as None, and the build reports success with a 0-node graph. Handled in medhypergraph/inference/extraction.py.

Reproduce the headline result

export CUDA_VISIBLE_DEVICES=""         
export MEDHG_LLM_MODEL=Qwen3.6-AWQ
export TOKENIZERS_PARALLELISM=false

# The published Qwen MedHyperGraph cell
medhg --backbone qwen35 eval boolean \
  --questions <ehrxqa boolean split>.json \
  --study-map question_study_map.json \
  --sample-size 0 --no-grounding --resume \
  --output runs/qwen_medhypergraph.json

A full 1900-question Qwen run takes roughly 5–6 hours. Results are written after every question, so --resume continues a killed run rather than restarting it, which also makes it safe to run detached:

nohup medhg --backbone qwen35 eval boolean ... > runs/qwen.log 2>&1 &

The verification run above gates on a band, never a point value, because of the noise floor: accuracy in [71.9, 75.8] for the full row and [62.0, 66.3] for --drop-ehr.

The other rows

# Table 1, Vanilla: no retrieval at all
medhg --backbone qwen35 eval boolean ... --no-retrieval --output runs/qwen_vanilla.json

# Table 3 ablations: one flag each
medhg --backbone qwen35 eval boolean ... --drop-ehr           --output runs/qwen_no_ehr.json
medhg --backbone qwen35 eval boolean ... --drop-reports       --output runs/qwen_no_reports.json
medhg --backbone qwen35 eval boolean ... --no-temporal-filter --output runs/qwen_no_temporal.json
medhg --backbone qwen35 eval boolean ... --drop-ontology      --output runs/qwen_no_ontology.json
medhg --backbone qwen35 eval boolean ... --grounding          --output runs/qwen_grounding.json

---

Web UI

The evidence viewer from the demo video: ask a question about a patient and watch which parts of their record the retriever actually used (prior reports with the cited span highlighted, EHR rows, the study timeline, and locSAM3 segmentation overlays on the chest X-rays.)

cd webui

# 1. Configure. Every setting is an environment variable.
cp .env.example .env      # then edit the four data roots and MEDHG_CONFIG
export $(grep -v '^#' .env | xargs)

# 2. Backend (FastAPI). The port must match the /api proxy target in
#    frontend/vite.config.js, which ships as 8082.
cd backend && pip install -r requirements.txt
uvicorn main:app --port 8082

# 3. Frontend (Vite dev server), in a second shell
cd webui/frontend && npm install && npm run dev

Then open http://localhost:5173.

port 8080 is where the inference server runs.

Variable	Meaning
MEDHG_GRAPH_ROOT	built per-patient graphs, one p<subject_id>/ per patient
MEDHG_REPORTS_ROOT	MIMIC-CXR free-text reports
MEDHG_IMAGES_ROOT	MIMIC-CXR-JPG images
MEDHG_EHR_ROOT	per-patient EHR tables
MEDHG_CONFIG	a configs/default.yaml for the retrieval stack
MEDHG_LLM_URL	chat-completions endpoint, default http://127.0.0.1:8080/v1/chat/completions
MEDHG_LLM_MODEL	served model name

The retrieval it displays goes through the same package as the evaluation, but its chat prompt (webui/backend/prompts_chat.py) is its own and is unrelated to the fixed evaluation prompt. The UI is a research demo not a clinical tool, and not part of any reported number. Full detail in webui/README.md.

---

Repository layout

src/medhypergraph/
  graph/        §2.2 hypergraph construction: the ingest stages
  retrieval/    §2.3 temporally aware retrieval, the temporal window, leakage control
  inference/    the answer prompt and the vLLM client
  judge/        §3 the two open-ended judges and their shared rubric
  eval/         the boolean and open-ended evaluation loops
  analysis/     §4 tables, metrics and confidence intervals
  grounding/    optional, λ_vis = 0
  data/         questions, images, the study map
configs/        one YAML plus a per-backbone overlay
third_party/    the frozen HyperGraphRAG fork; §2.3 executes inside it
webui/          the evidence viewer from the supplementary video

third_party/hypergraphrag/ is a fork of LHRLAB/HyperGraphRAG at a804827, diverging by 3,331 lines. Paper §2.3 executes inside it, so it is kept behaviour-frozen: AST-identical to the copy that produced the numbers, with byte differences confined to three enumerated comment redactions. Every modification is catalogued in third_party/hypergraphrag/PATCHES.md.

Nothing under src/ hardcodes an absolute path. configs/default.yaml is the only file that changes per machine.

Licence

MIT for the code in this repository, see LICENSE. third_party/hypergraphrag/ retains upstream's MIT licence.

MIMIC-CXR-JPG, MIMIC-CXR, MIMIC-IV, EHRXQA and Chest ImaGenome are PhysioNet credentialled-access datasets, are not covered by this licence, and are not redistributed here.