Are all leaderboard results evaluated under 0-shot?

[zhimin-z]

https://opencompass.org.cn/leaderboard-llm

[yingfhu]

No, the n-shot setting for each dataset is different, you can click the score of the specific dataset on leaderboard to check its configuration. It will tell whether its 0-shot or n-shot from config file.

[yingfhu]

The n-shot might be implicit, you need to read the config especially the content in retriever and prompt_template filed. In your example, the nq is tested on three settings 0-shot, 1-shot and 5-shot.

[yingfhu]

This one is special, normally we won't use multiple shot settings in one config. The nq score on leaderboard is 0-shot

[zhimin-z]

Thanks for the quick replies!

How do you know that this nq is tested on three settings 0-shot, 1-shot, and 5-shot? And how do you know the leaderboards is 0-shot rather than the others?

Due to time constraints, I am a bit lost in the code snippet that I should read...Any trick that can save me time?

Thanks in advance!

[yingfhu]

0-shot :

retriever=dict(type=ZeroRetriever),

k-shot :

retriever=dict(type=FixKRetriever, fix_id_list=list(range(k))),

use 0,1,5 settings

for k in [0, 1, 5]: xxxxx

We use 0-shot for nq in leaderboard is not shown, i mentioned that it is a special case, normally we won't use multiple shot settings in one config.