Welcome to MedAlign Discussions!

[scottfleming]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded.

To get started, comment below with an introduction of yourself and tell us about your questions and ideas.

[alexottenwess]

Hey @scottfleming, really great paper. I'm doing some research on using LLMs to retrieve and summarize FHIR records. I'm really interested in getting my hands on the question set and category mappings discussed in your paper. Are these going to be open-sourced anytime soon? I don't need the actual EHR data itself, just the questions and ideally mappings to categories. Let me know what you think! Thanks

[scottfleming]

Hi @alexottenwess thanks so much for the feedback! We're looking to release the full instruction set, their category mappings, and the EHRs (under a DUA) all together within the next several months.

That being said, I've put the instruction taxonomy we used in the repo:
https://github.com/som-shahlab/medalign/blob/main/data/instruction-taxonomy.csv
Note that the instruction taxonomy was built by a practicing physician, but we don't know what inter-rater reliability would look like on assigning instructions to categories. GPT-4 gets about ~60% "accuracy" in the zero-shot setting assigning instructions to categories, assuming human labels are ground truth.

And here 60+ instructions that we included in the paper in a .csv for easy processing and analysis:
https://github.com/som-shahlab/medalign/blob/main/data/instructions-sample.csv

The following file (included in the supplement of the paper, but also uploaded to the repo for easy downloading and inspection) contains a synthetic but representative EHR very similar to the kind of EHRs we passed into the LLMs in our experiments:
https://github.com/som-shahlab/medalign/blob/main/data/sample-ehr-clean.xml

Hopefully this allows you to get started.

[starmpcc]

Hello,

I have a couple of questions regarding the dataset:

1. As the structured EHR data is currently in XML format, would it be possible to provide the schema information of the source tables? I am interested in converting them to a database format, which would facilitate the use of Text-to-SQL models for analyzing the dataset.

2. Regarding access to the dataset, is it possible to obtain early access by providing my PhysioNet credentials or CITI certification via email? If this is not feasible, could you please provide an estimated timeline for the dataset's public release?

Thank you