ChallengeClinicalQA

Repo for the paper Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

Datasets

• Medbullets
• JAMA Clinical Challenge

We do not publicably release the JAMA Clinical Challenge data due to license constraints. Instead, we provide URLs to the articles and a scraper that you can use to obtain the data with the appropriate license. Please check your license to ensure you have access to JAMA articles (Full Text) before you run the script.

Install the required dependencies

pip install -r requirements.txt

Scrape the data

python jama_scraper.py

The data will be saved in jama_raw.csv and jama_raw.json files.

Reference

If you find this repository helpful, please cite our paper:

@article{chen2024benchmarking,
  title={Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions},
  author={Chen, Hanjie and Fang, Zhouxiang and Singla, Yash and Dredze, Mark},
  journal={arXiv preprint arXiv:2402.18060},
  year={2024}
}