leval and longbench issue

[nlpcat]

what would be the suggested arguments for leval and longbench
--max-seq-len
--max-out-len

[nlpcat]

python run.py --datasets leval_coursera_gen --hf-path THUDM/chatglm2-6b-32k --model-kwargs device_map='auto' trust_remote_code=True torch_dtype='auto' --tokenizer-kwargs padding_side='left' truncation='left' use_fast=False trust_remote_code=True --num-gpus 1 --debug --batch-size 1 --no-batch-padding --max-seq-len 32000 --max-out-len 256

this can't reproduce the result 45.35 here.

machine is a100 and result is 36.05

dataset         version    metric    mode      opencompass.models.huggingface.HuggingFace_THUDM_chatglm2-6b-32k
--------------  ---------  --------  ------  ------------------------------------------------------------------
LEval_coursera  36a006     accuracy  gen                                                                  36.05

https://github.com/open-compass/opencompass/blob/main/docs/en/advanced_guides/longeval.md#long-context-ability-ranking

used the latest commit. 9db5652

[philipwangOvO]

Here is part of my config in evaluation.

tokenizer_kwargs=dict(
     padding_side='left',
     truncation_side='left',
     trust_remote_code=True,
),
max_out_len=1024,
max_seq_len=32768,
batch_size=1,
model_kwargs=dict(trust_remote_code=True, device_map='auto'),
mode='mid',
run_cfg=dict(num_gpus=1, num_procs=1),

The argument max_seq_len is dependent on the maximum length of chosen model, and the argument max_out_len is dependent on the task.
Please note that when evaluating on L-Eval and LongBench, make sure to set the mode to 'mid'. It will truncate the input if its length exceeds max_seq_len.
You can also refer to huggingface.py for more information.

[nlpcat]

@philipwangOvO I tried this config but there was with no change on the result. it is still 36.05

[philipwangOvO]

Please check your python version.
I've tried python 3.9.17 and python 3.10.13, it turns out to be both 45.35.

[lxy0727]

Hi, I ran into the same issue, and here is my config:

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='chatglm2-6b-32k',
        path='THUDM/chatglm2-6b-32k',
        batch_padding=False,
        model_kwargs=dict(device_map='auto'),
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left', use_fast=False),
        max_seq_len=32768,
        max_out_len=1024,
        batch_size=1,
        run_cfg=dict(num_gpus=1, num_procs=1),
        mode='mid',
    )
]

would you mind sharing the full config and the version of opencompass? Really appreciate it, thank you :)