Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api/datasets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ Available Datasets
datasets/pyhealth.datasets.BMDHSDataset
datasets/pyhealth.datasets.COVID19CXRDataset
datasets/pyhealth.datasets.ChestXray14Dataset
datasets/pyhealth.datasets.VQARADDataset
datasets/pyhealth.datasets.TUABDataset
datasets/pyhealth.datasets.TUEVDataset
datasets/pyhealth.datasets.ClinVarDataset
Expand Down
11 changes: 11 additions & 0 deletions docs/api/datasets/pyhealth.datasets.VQARADDataset.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
pyhealth.datasets.VQARADDataset
===================================

The VQA-RAD dataset for medical visual question answering. The dataset loader
converts the public JSON annotations into a flat metadata CSV that PyHealth can
ingest, and its default task is :class:`~pyhealth.tasks.MedicalVQATask`.

.. autoclass:: pyhealth.datasets.VQARADDataset
:members:
:undoc-members:
:show-inheritance:
26 changes: 21 additions & 5 deletions docs/api/models/pyhealth.models.MedFlamingo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,37 @@ pyhealth.models.MedFlamingo

MedFlamingo: multimodal medical few-shot learner.

The separate callable MedFlamingoLayer (gated cross-attention dense block)
and the complete MedFlamingo model.
This reference covers the visual resampler, the gated cross-attention
building block, and the complete MedFlamingo model used in the VQA-RAD
integration branch.

**Paper:** Moor et al. "Med-Flamingo: a Multimodal Medical Few-shot Learner" ML4H 2023.

.. note::

This is a stub implementation. The class structure and signatures are
in place, but ``forward()`` and ``generate()`` raise ``NotImplementedError``.
``forward()`` follows the PyHealth training contract for dataset-backed
classification-style use, while ``generate()`` provides the multimodal
prompting path for direct medical VQA generation.

.. autoclass:: pyhealth.models.MedFlamingoLayer
PerceiverResampler
------------------

.. autoclass:: pyhealth.models.medflamingo.PerceiverResampler
:members:
:undoc-members:
:show-inheritance:

MedFlamingoLayer
----------------

.. autoclass:: pyhealth.models.medflamingo.MedFlamingoLayer
:members:
:undoc-members:
:show-inheritance:

MedFlamingo
-----------

.. autoclass:: pyhealth.models.MedFlamingo
:members:
:undoc-members:
Expand Down
1 change: 1 addition & 0 deletions docs/api/tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ Available Tasks
DKA Prediction (MIMIC-IV) <tasks/pyhealth.tasks.dka>
Drug Recommendation <tasks/pyhealth.tasks.drug_recommendation>
Length of Stay Prediction <tasks/pyhealth.tasks.length_of_stay_prediction>
Medical VQA <tasks/pyhealth.tasks.MedicalVQATask>
Medical Transcriptions Classification <tasks/pyhealth.tasks.MedicalTranscriptionsClassification>
Mortality Prediction (Next Visit) <tasks/pyhealth.tasks.mortality_prediction>
Mortality Prediction (StageNet MIMIC-IV) <tasks/pyhealth.tasks.mortality_prediction_stagenet_mimic4>
Expand Down
12 changes: 12 additions & 0 deletions docs/api/tasks/pyhealth.tasks.MedicalVQATask.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
pyhealth.tasks.MedicalVQATask
===================================

Medical visual question answering task for paired radiology images and
questions. This task treats VQA-RAD answers as a multiclass prediction target
so the resulting ``SampleDataset`` can be trained with the standard PyHealth
trainer loop.

.. autoclass:: pyhealth.tasks.MedicalVQATask
:members:
:undoc-members:
:show-inheritance:
Loading