Llama-3.1-8B-Instruct_math shows poor performance on GSM8K-CoT

[tanganke]

Congratulations on the release of MergeBench!

While testing the model on gsm8k_cot, I noticed that the math ability of Llama-3.1-8B-Instruct_math is appears to be quite poor.

The following evaluation was run using lm_eval:

lm_eval \
    --model_args pretrained="MergeBench/Llama-3.1-8B-Instruct_math",dtype='bfloat16',parallelize=True \
    --apply_chat_template \
    --tasks gsm8k_cot \
    --batch_size 8

Results:

| Tasks     | Version | Filter           | n-shot | Metric       | Value   | Stderr   |
|-----------|---------|------------------|--------|--------------|---------|----------|
| gsm8k_cot | 3       | flexible-extract | 8      | exact_match  | 0.1698  | ± 0.0103 |
|           |         | strict-match     | 8      | exact_match  | 0.0000  | ± 0.0000 |

[Zheng-Jay]

Congratulations on the release of MergeBench!恭喜 MergeBench 发布！

While testing the model on gsm8k_cot, I noticed that the math ability of Llama-3.1-8B-Instruct_math is appears to be quite poor.在 gsm8k_cot 上测试模型时，我注意到 Llama-3.1-8B-Instruct_math 的数学能力似乎很差。

The following evaluation was run using lm_eval:使用 lm_eval 运行以下评估：

lm_eval
--model_args pretrained="MergeBench/Llama-3.1-8B-Instruct_math",dtype='bfloat16',parallelize=True
--apply_chat_template
--tasks gsm8k_cot
--batch_size 8
Results:  结果：

Tasks	Version	Filter	n-shot	Metric	Value	Stderr
gsm8k_cot	3	flexible-extract	8	exact_match	0.1698	± 0.0103
		strict-match	8	exact_match	0.0000	± 0.0000

I also encountered the same situation, and my score is even lower than that of Llama-3.2-1B-Instruct (value = 0.3442).

[yifei-he]

Sorry for the late reply as this work was previously going through a reviewing phase. Since the review process is single-blinded and this repo is public, we unfortunately cannot engage in discussions per the conference policy. Now that the decision is out, I can work on resolving the issues.

The problem might come from the rigid answer extraction template provided in lmeval. We recommend doing the following modification: In lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k-cot.yaml, change regex_pattern: The answer is (\-?[0-9\.\,]+). to regex_pattern: The answer is(?:\s*:)?\s*(?:####\s*)?(?:\$\\?boxed\{|\$)?\s*(-?[0-9,\.]+)\s*(?:\}?\$)?\.? This would account for more possible formatting, and extract the answer in a more flexible manner.