如何单独进行infer和eval

[IcyFeather233]

我有一个需求是我想添加一个API model来做eval_subjective_alignbench的评测，我想让这个评测infer和eval分开跑，即我先跑出来infer结果，然后慢慢改model文件来跑eval。我想知道怎么做到指定之前某次的infer结果单独跑eval?

[DseidLi]

Hi, you can try

python run.py path/to/your/config.py -r path/to/infer/folder --mode eval

like this

python run.py path/to/your/config.py -r 20240117_XXXXXX --mode eval

[IcyFeather233]

workspace/opencompass$ HF_EVALUATE_OFFLINE=1 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python run.py configs/eval_my_alignbench.py -m eval --debug --resume outputs/alignment_bench/20240403_052257
usage: run.py [-h] [--slurm | --dlc] [--mm-eval] [--models MODELS [MODELS ...]] [--datasets DATASETS [DATASETS ...]] [--summarizer SUMMARIZER]
              [--debug] [--dry-run] [-m {all,infer,eval,viz}] [-r [REUSE]] [-w WORK_DIR] [--config-dir CONFIG_DIR] [-l]
              [--max-partition-size MAX_PARTITION_SIZE] [--gen-task-coef GEN_TASK_COEF] [--max-num-workers MAX_NUM_WORKERS]
              [--max-workers-per-gpu MAX_WORKERS_PER_GPU] [--retry RETRY] [--dump-eval-details] [-p PARTITION] [-q QUOTATYPE] [--qos QOS]
              [--aliyun-cfg ALIYUN_CFG] [--hf-path HF_PATH] [--peft-path PEFT_PATH] [--tokenizer-path TOKENIZER_PATH]
              [--model-kwargs MODEL_KWARGS [MODEL_KWARGS ...]] [--tokenizer-kwargs TOKENIZER_KWARGS [TOKENIZER_KWARGS ...]]
              [--max-out-len MAX_OUT_LEN] [--max-seq-len MAX_SEQ_LEN] [--no-batch-padding] [--batch-size BATCH_SIZE] [--num-gpus NUM_GPUS]
              [--pad-token-id PAD_TOKEN_ID] [--custom-dataset-path CUSTOM_DATASET_PATH] [--custom-dataset-meta-path CUSTOM_DATASET_META_PATH]
              [--custom-dataset-data-type {mcq,qa}] [--custom-dataset-infer-method {gen,ppl}]
              [config]
run.py: error: unrecognized arguments: --resume outputs/alignment_bench/20240403_052257

[IcyFeather233]

It seems that --resume should be changed to -r, and I think the path/to/infer/folder should be like 20240403_052257, not outputs/default/20240117_XXXXXX, or it will shows AssertionError: No predictions found.

[DseidLi]

Thank you for correcting my command line; I've updated my reply based on your accurate feedback.