[LLaMA Adapter V2] Evaluation on multiple choice questions.

[xiujiesong]

Hi,

Thanks for your great work. I am wondering how can we evaluate LLaMA Adapter V2 on multiple choice questions. I expect the model can generate the letter (A/B/C/D) of the corresponding choice, but it often does not generate the letters even if I tell it to do so. Do you have any good solutions?

[csuhan]

hi @xiujiesong ，I think you can construct a prompt like:
[Question] A. xxx. B. xxx. C. xxx. D. xxx. Answer: [A/B/C/D].

Then you can test the loss of each answer (similar to the approach of Perplexity) and choose the one with lowest loss.