Measurement: expert utilisation in minimind-3-moe (routing is ~3.8x more imbalanced than random init)

[JustinGuese]

I measured expert utilisation in the released minimind-3-moe and wanted to share the numbers, because one of them surprised me and I could not find this reported anywhere.

Short version: routing in the trained model is ~3.8× more imbalanced than the same architecture at random initialisation, and 5 of the 32 expert slots receive under 2.5% of tokens each. This is a measurement, not a bug report — expert imbalance is not automatically bad, and I say what I think it does and does not show at the end.

Method

The released checkpoint exports as a stock Qwen3MoeForCausalLM, so this needs no minimind code — just a forward hook on each layer's mlp.gate, capturing the full router softmax and the top-1 assignment.

• Data: 1346 questions from C-Eval (ceval/ceval-exam, val split, all 52 subjects, shuffled so a shard is not one topic). C-Eval was chosen because it is already one of the benchmarks in the README table.
• Split into 5 disjoint shards so the metric has a measured noise floor. A load-imbalance number without a floor is not interpretable.
• Metrics, chosen for E=4 / top-1:
  • load_tv = ½·L1(normalised per-expert token counts, uniform). 0 = perfectly uniform, 0.75 = everything on one expert at E=4.
  • Router entropy over the full softmax, max = ln 4 = 1.386 nats. (A top-1-vs-top-2 binary entropy would discard a third of the distribution at E=4.)
  • I deliberately did not use union-sparsity (1 − n_active/E): at E=4 it saturates at 0 for any batch ≥ 8 and carries no information.
• Control: the same config instantiated with random weights (from_config, no pretrained load), same data, same everything. This is the part that makes the number interpretable — it says what routing looks like before training.

Result

	random init	trained minimind-3-moe
load_tv (0 = uniform, 0.75 = collapsed)	0.068	0.255
router entropy	1.280 nats (92.3% of max)	0.863 (62.3%)
expert slots receiving < 2.5% of tokens	0 / 32	5 / 32
noise floor (same-layer, across shards)	0.017	0.006

The trained imbalance is 41× the shard-to-shard noise floor, so it is not sampling variation. Per layer:

layer	load_tv ± shard std	entropy (nats)	per-expert share
0	0.316 ± 0.010	1.031	0.401 / 0.016 / 0.168 / 0.415
1	0.235 ± 0.006	1.096	0.428 / 0.243 / 0.307 / 0.022
2	0.142 ± 0.004	0.767	0.188 / 0.392 / 0.192 / 0.228
3	0.191 ± 0.006	0.888	0.322 / 0.361 / 0.059 / 0.258
4	0.169 ± 0.006	0.853	0.229 / 0.276 / 0.393 / 0.103
5	0.264 ± 0.005	0.824	0.502 / 0.215 / 0.021 / 0.262
6	0.416 ± 0.008	0.756	0.648 / 0.081 / 0.002 / 0.268
7	0.306 ± 0.006	0.691	0.014 / 0.381 / 0.424 / 0.180

Layer 6 is the extreme: one expert takes 64.8% of tokens while another takes 0.22%. Uniform would be 25% each.

Note also that entropy decreases monotonically with depth (1.03 → 0.69), i.e. the router grows more confident in later layers, while imbalance is worst at the first and last layers.

What I think this does and does not show

Does not show: that this is a bug, or that quality is harmed. Some imbalance is expected and can reflect genuine specialisation; I have not measured any downstream effect, and I am not claiming one.

Does show, I think, one thing worth considering: with num_experts=4, five slots sitting under 2.5% means the effective number of experts is meaningfully below four. That interacts with cost — at moe_intermediate_size == intermediate_size (which is what the released config has, 2432 = 2432), 4 experts cost 4× the dense FFN parameters for the same active parameters at top-1. If one or two of those experts are close to unused, the parameters are being paid for without being used.

A possible follow-up, if this is interesting to you: router_aux_loss_coef is 5e-4 in MiniMindConfig but 0.001 in the exported config.json — I do not know which produced the released weights, and whether a larger coefficient flattens this would be a cheap experiment.

Reproducing

Self-contained script (hooks, metrics, shard noise floor, random-init control): I am happy to attach it or open a PR adding it under scripts/ if you would find that useful. It needs only transformers + datasets and runs on CPU in a few minutes.

python routing_audit.py --docs 2000 --shards 5              # trained
python routing_audit.py --docs 2000 --shards 5 --random_init # control

Caveats stated plainly: one data distribution only (Chinese exam questions — routing may differ on other text); one checkpoint; measured on the val split; and the random-init control uses a single seed, so its 0.068 is one draw, not a distribution.

---

---

我测了一下已发布的 minimind-3-moe 的专家利用率，想把数据分享出来，因为其中一项结果出乎我的意料，而且我没有找到别处报告过。

结论先说：训练后的路由比同架构随机初始化时失衡约 3.8 倍，32 个专家槽里有 5 个各自只拿到不到 2.5% 的 token。 这是一次测量，不是 bug 报告——专家负载不均本身并不必然是坏事，文末我会说明我认为它能说明什么、不能说明什么。

方法

发布的 checkpoint 导出为标准的 Qwen3MoeForCausalLM，所以完全不需要 minimind 的代码——只要在每层的 mlp.gate 上挂一个前向钩子，取完整的 router softmax 和 top-1 分配。

• 数据：C-Eval（ceval/ceval-exam，val 划分，全部 52 个学科，打乱以免某个分片只包含一个学科）共 1346 道题面。选 C-Eval 是因为它本来就是 README 表格里的基准之一。
• 切成 5 个互不重叠的分片，让指标有一个实测的噪声地板。没有地板的负载不均衡数值是无法解释的。
• 指标（针对 E=4 / top-1 选择）：
  • load_tv = ½·L1(归一化的每专家 token 数, 均匀分布)。0 = 完全均匀，E=4 时 0.75 = 全部挤在一个专家。
  • 完整 softmax 的熵，上界 ln 4 = 1.386 nats。（top-1 vs top-2 的二元熵在 E=4 时会丢掉三分之一的分布。）
  • 我特意没有用 union sparsity（1 − n_active/E）：E=4 且 batch ≥ 8 时它恒为 0，没有任何信息量。
• 对照组：同一份 config 随机初始化（from_config，不加载预训练权重），数据与其余设置完全相同。正是这一项让数值变得可解释——它给出了训练之前路由长什么样。

结果

	随机初始化	训练后的 minimind-3-moe
load_tv（0 = 均匀，0.75 = 塌缩）	0.068	0.255
router 熵	1.280 nats（上界的 92.3%）	0.863（62.3%）
占比 < 2.5% 的专家槽	0 / 32	5 / 32
噪声地板（同层跨分片 std）	0.017	0.006

训练后的失衡是分片噪声地板的 41 倍，因此不是抽样波动。逐层数据见上方英文表格；第 6 层最极端：一个专家拿走 64.8% 的 token，另一个只有 0.22%（均匀应为各 25%）。另外，熵随深度单调下降（1.03 → 0.69），即越靠后的层 router 越"自信"，而失衡在首层和末层最严重。

我认为它说明什么、不说明什么

不能说明：这是一个 bug，或者质量因此受损。一定程度的不均衡是预期之内的，也可能反映真实的专业化分工；我没有测量任何下游影响，也不声称存在这种影响。

我认为能说明一件值得考虑的事：在 num_experts=4 的前提下，有五个槽位低于 2.5%，意味着有效专家数明显不足 4。这与成本相关——在 moe_intermediate_size == intermediate_size 时（已发布的 config 正是如此，2432 = 2432），4 个专家的 FFN 参数量是 dense 的 4 倍，而 top-1 下的激活参数量与 dense 相同。如果其中一两个专家几乎不被使用，那这部分参数就是付了钱却没有被用上。

一个可能的后续（如果您觉得有意思）：router_aux_loss_coef 在 MiniMindConfig 里是 5e-4，而导出的 config.json 里是 0.001；我不确定发布的权重是用哪个训出来的，而"加大这个系数能否把分布压平"是一个成本很低的实验。

复现

脚本是自包含的（钩子、指标、分片噪声地板、随机初始化对照），只依赖 transformers + datasets，CPU 上几分钟即可跑完。如果您觉得有用，我可以附上，或者提一个 PR 放到 scripts/ 下。

需要如实说明的局限：只测了一种数据分布（中文考试题面，其他文本上的路由可能不同）；只测了一个 checkpoint；用的是 val 划分；随机初始化对照只用了一个种子，所以那个 0.068 是单次抽样而不是一个分布。