如何在进行few-shot的评估啊，比如5-shot，在baichuan模型上

[xealml]

如题，翻了下官方文档，似乎没有明确的引导

[yingfhu]

Most of our fewshot example are fixed inside the prompt template of each dataset config , you could refer to https://opencompass.readthedocs.io/zh-cn/latest/prompt/prompt_template.html for details.

The dynamic few-shot are under https://github.com/open-compass/opencompass/blob/main/opencompass/openicl/icl_retriever, the documents will be added in the future, you could refer to the docstring of each retriever for details.

[xealml]

还想问下使用如下的默认方式，是zero-shot的还是N-shot呢

run.py --datasets ceval_ppl cmmlu_ppl \
  --hf-path ./Baichuan2-7B-Base \
  --tokenizer-path ./Baichuan2-7B-Base \
  --model-kwargs device_map='auto' trust_remote_code=True torch_dtype=torch.bfloat16 \
  --tokenizer-kwargs use_fast=False trust_remote_code=True \
  --max-out-len 100 \
  --max-seq-len 4096 \
  --batch-size 8 \
  --no-batch-padding \
  --num-gpus 1

[xealml]

还想问下使用如下的默认方式，是zero-shot的还是N-shot呢

run.py --datasets ceval_ppl cmmlu_ppl \
  --hf-path ./Baichuan2-7B-Base \
  --tokenizer-path ./Baichuan2-7B-Base \
  --model-kwargs device_map='auto' trust_remote_code=True torch_dtype=torch.bfloat16 \
  --tokenizer-kwargs use_fast=False trust_remote_code=True \
  --max-out-len 100 \
  --max-seq-len 4096 \
  --batch-size 8 \
  --no-batch-padding \
  --num-gpus 1

看了下源码，应该是固定的5-shot的模式，不知道我理解的对不对

[yingfhu]

Yes，you can use python tools/list_configs.py to check which config file is using, and the dataset config shows whether it use n-shots. You should read the config file. The defaults for different datasets are different.