Measurement: error bars on the objective benchmark table (0 of 6 dense-vs-MoE differences are significant)

[JustinGuese]

I re-ran the objective benchmark table to answer a question I could not answer from the README: how large does a difference in that table have to be before it means something?

lm_eval reports a standard error for every metric, but the published table shows point estimates only. Below are the error bars. Short version: the instrument's resolution is roughly ±0.5 to ±2 percentage points per task, and none of the six dense-vs-MoE differences in the table are distinguishable from zero at that resolution.

This is not a claim that anything is wrong. It is a claim about what the numbers can and cannot support, and I think it makes the table more useful rather than less.

First, reproduction

I could reproduce the table once I worked out the exact protocol, which took a couple of tries and is worth writing down:

• The published numbers use acc_norm where the task provides it, and acc otherwise (social_iqa, ceval).
• They were produced without --apply_chat_template — even though the command in README.md:1571-1584 includes that flag. With the flag, arc_easy reads 35.40 instead of 28.49; the gap is entirely the flag.

With that protocol (lm_eval 0.4.13, minimind-3 / minimind-3-moe from HF, dtype=auto):

task	published	my reproduction	deviation
ceval-valid	24.89	24.29 ± 1.17	0.60
arc_easy	28.49	28.37 ± 0.92	0.12
piqa	50.65	51.03 ± 1.17	0.38
openbookqa	23.60	25.80 ± 1.96	2.20
hellaswag	28.28	27.82 ± 0.45	0.46
social_iqa	34.19	33.42 ± 1.07	0.77

Every deviation is within about one standard error. The table reproduces.

(Two practical notes: cmmlu no longer loads at all on current datasets — it ships a cmmlu.py loading script and datasets ≥ 4 refuses those — so the documented command fails as written and I had to drop that column. And since the flag in the documented command was not the one used, anyone following the README will get different numbers than the table shows.)

The error bars

task	metric	± se
hellaswag	acc_norm	±0.45
arc_easy	acc_norm	±0.92
social_iqa	acc	±1.07
ceval-valid	acc_norm	±1.17
piqa	acc_norm	±1.17
openbookqa	acc_norm	±1.96

These follow directly from the number of items per task (openbookqa has 500; hellaswag ~10k).

What that implies for the dense-vs-MoE comparison

For a difference between two models, the standard error is roughly √(se₁² + se₂²). Applying that to the published rows:

task	dense	MoE	Δ	se(Δ)	Δ/se
ceval-valid	24.89	25.48	+0.59	1.65	0.36
arc_easy	28.49	27.74	−0.75	1.31	0.57
piqa	50.65	50.71	+0.06	1.65	0.04
openbookqa	23.60	26.20	+2.60	2.78	0.94
hellaswag	28.28	27.43	−0.85	0.63	1.34
social_iqa	34.19	34.03	−0.16	1.52	0.11

Zero of six reach 1.96σ. The largest is 1.34σ. So on this evidence the 64M dense model and the 198M MoE model are not separated by these benchmarks in either direction — which is a legitimate and interesting result for a teaching repo, and arguably a more honest headline than any individual row.

Separately: C-Eval is four-choice, so the random baseline is 25.00. The published values are 24.89 and 25.48 with se ≈ 1.17. Both are at chance. The same is true of the CMMLU column in the published table (24.32–25.38). That does not mean the models are broken — models this size genuinely are at chance on these benchmarks — but it does mean those two columns cannot rank anything, including against the third-party baselines in the same table, several of which are also in the 24–25 band.

One thing I want to be careful not to overclaim

The minimind-3-exam result (README.md:1609, "平均提升约 2.9 个百分点") is in a different statistical position and I am not lumping it in. It is an average over 7 tasks, and averaging shrinks the error bar by roughly √7, so an average uplift of ~2.9pp against per-task errors of ~1pp is plausibly real even though no single row would be. I have not measured the exam checkpoint, and I am not casting doubt on it — I mention it only to be explicit that "these error bars are large" is a per-task statement, not a blanket one.

What I'd suggest

Only this: publish the stderr column alongside the table, plus the harness version and the exact command used. lm_eval already emits all of it in its results JSON. That costs nothing, and it lets a reader tell at a glance which comparisons in the table are meaningful.

I am happy to open a PR adding the ± column and a note on the protocol, if that would be welcome. Raw results JSON for both models is available if you want to check any of the above.

What this does not cover

The error bars above are evaluation error only — they describe sampling over benchmark items for a fixed checkpoint. They say nothing about training variance, i.e. how much the numbers would move if the same config were trained again under a different seed. That is likely the larger source of variation and it is currently unmeasurable from outside, because the trainer seed is hardcoded (setup_seed(42 + rank)); #836 adds a --seed flag which would make such a study possible. I have not run it — it needs 5 training runs per variant — so I want to be clear that this post measures only one of the two noise sources.

---

---

我把客观评测表重跑了一遍，想回答一个从 README 里无法回答的问题：那张表里的差异要多大，才算真的有意义？

lm_eval 对每个指标都会给出标准误，但已发布的表格只有点估计。下面是误差棒。结论先说：这套评测的分辨率大约是每个任务 ±0.5 到 ±2 个百分点，而表中 dense 与 MoE 的六项差异，没有一项能在这个分辨率下与 0 区分开。

这不是说哪里有错。这是关于"这些数字能支持什么、不能支持什么"的说明，我认为它会让这张表更有用而不是更没用。

首先是复现

我确实复现出了这张表，但摸清确切口径花了几次尝试，值得记下来：

• 已发布的数字在任务提供 acc_norm 时用 acc_norm，否则用 acc（social_iqa、ceval 用 acc）。
• 它们是在不加 --apply_chat_template 的情况下跑出来的——尽管 README.md:1571-1584 里的命令带了这个参数。加上它之后，arc_easy 会变成 35.40 而不是 28.49，差距完全来自这个参数。

按这个口径（lm_eval 0.4.13，HF 上的 minimind-3 / minimind-3-moe，dtype=auto）复现结果见上方英文表格：每一项的偏差都在约一个标准误以内，表格可以复现。

（两个实际问题：cmmlu 在当前的 datasets 上已经完全加载不了——它带一个 cmmlu.py 加载脚本，而 datasets ≥ 4 不再支持脚本——所以文档里的命令按原样是跑不通的，我只能去掉这一列。另外，既然文档命令里的参数与实际使用的口径不一致，照着 README 跑的人得到的数字会和表格对不上。）

误差棒

见上方英文表格：hellaswag ±0.45，arc_easy ±0.92，social_iqa ±1.07，ceval-valid ±1.17，piqa ±1.17，openbookqa ±1.96。这直接由每个任务的样本量决定（openbookqa 只有 500 题，hellaswag 约 1 万题）。

这对 dense vs MoE 的比较意味着什么

两个模型之差的标准误约为 √(se₁² + se₂²)。代入已发布的数值（表见上方英文部分）：六项中没有一项达到 1.96σ，最大的一项是 hellaswag 的 1.34σ。也就是说，就这份证据而言，64M 的 dense 模型与 198M 的 MoE 模型在这些基准上并没有被区分开——对一个教学仓库来说这本身是个合理且有意思的结论，而且可以说比其中任何单独一行都更诚实。

另外：C-Eval 是四选一，随机基线是 25.00。 已发布的两个值是 24.89 和 25.48，se ≈ 1.17，都处于随机水平；表中 CMMLU 一列（24.32–25.38）同理。这并不意味着模型有问题——这个规模的模型在这些基准上本来就处于随机水平——但它确实意味着这两列无法给任何东西排序，包括同表中那几个同样落在 24–25 区间的第三方基线。

有一点我想特别避免过度解读

minimind-3-exam 的结论（README.md:1609，"平均提升约 2.9 个百分点"）在统计上处于不同的位置，我没有把它归为一类。它是 7 个任务的平均，而平均会把误差棒缩小约 √7 倍，所以在每任务约 1pp 的误差下，约 2.9pp 的平均提升是很可能真实的，尽管任何单独一行都达不到显著。我没有测过 exam 这个 checkpoint，也不是在质疑它——特意提一句，是为了说明"误差棒很大"是针对单个任务的陈述，不是一竿子打翻。

我的建议

只有一条：把 stderr 一列和评测所用的 harness 版本、确切命令一起发布出来。 lm_eval 的结果 JSON 里本来就都有，成本为零，却能让读者一眼看出表中哪些比较是有意义的。

如果您觉得合适，我可以提一个 PR，把 ± 一列和口径说明加进去。两个模型的原始结果 JSON 我都留着，随时可以核对。

本文没有覆盖的部分

上面的误差棒只是评测误差——它描述的是在 checkpoint 固定的前提下，基准题目抽样带来的波动。它完全没有涉及训练方差，也就是同一份配置换个随机种子重训一遍，这些数字会移动多少。后者很可能是更大的一个方差来源，而目前从外部无法测量，因为 trainer 的种子是写死的（setup_seed(42 + rank)）；#836 加了 --seed 参数，这类实验才成为可能。我没有跑——那需要每个变体训练 5 次——所以想明确说明：本文只测了两个噪声来源中的一个。