About USMLE results

[chaoyi-wu]

Hello have you tried to fine-tune the llama-7B with the USMLE training set. May you please share the final results you get. I try to do this and get only around 30 ACC. I am wondering whether this is correct. Thanks!

[shizhediao]

Hi,
We have not fine-tuned the LLaMA-7B on the USMLE training data. It seems low with 30% accuracy on in-domain data. One possible reason is the training data of USMLE is small (around 10K), while the training data of PubMedQA and MedMCQA is about 400K although they have different data distributions (OOD).

Could you try our example first to see whether you could reproduce our results? We choose this setting because, in Galactica, they deem PubMedQA and MedMCQA as in-domain and USMLE as out-of-domain data.

Thanks!