Eval note: large MoE models may benefit from full reasoning replay

[antoinezambelli]

Forge defaults to reasoning_replay="none": captured reasoning remains available for observability, but it is not re-sent to the model on later tool turns.

That default came from our earlier replay sweep. Across the predominantly smaller, dense models available at the time, replaying prior reasoning increased context and token usage without producing a significant score improvement. none was and remains the more efficient general default.

The latest model sweep surfaced a new pattern worth sharing: every MoE model above 14B scored higher with full reasoning replay.

What we observed

In native-mode, reforged runs comparing full against none:

Model	Architecture	None	Full	Difference
Gemma-4 26B-A4B	MoE	82.23%	84.15%	+1.92 pp
Qwen3.6 35B-A3B	MoE	88.00%	95.15%	+7.15 pp
Qwen3.5 122B-A10B	MoE	91.31%	94.46%	+3.15 pp
Nemotron Super 120B-A12B	MoE	74.54%	81.08%	+6.54 pp
GPT-OSS 120B (~5.1B active)	MoE	74.15%	83.69%	+9.54 pp
GPT-OSS 120B, high effort	MoE	81.92%	90.85%	+8.92 pp

That is 5/5 MoE model families above 14B responding positively to full replay.

The three smaller MoEs did not show the same behavior:

Model	None	Full	Difference
LFM2.5 8B-A1B	57.69%	58.23%	+0.54 pp
Mellum2 12B-A2.5B Instruct	81.00%	80.77%	-0.23 pp
Mellum2 12B-A2.5B Thinking	68.38%	65.62%	-2.77 pp

Dense models were also largely insensitive. Qwen3.6-27B moved only +0.62 points; Gemma-4-31B was the notable exception at +2.23 points.

Each arm contains 1,300 runs: 26 scenarios x 50 repetitions. Score uses Forge's canonical definition, correct / total, so incomplete and invalid runs remain in the denominator. The large-MoE improvements were significant in the paired run-level comparisons.

What this means

This is an operational pattern in the current eval matrix, not a claim that MoE architecture universally requires full reasoning replay. Model size, reasoning behavior, serving templates, and training may all contribute.

It does mean replay policy should no longer be assumed to be quality-neutral for every model.

If you are deploying a large reasoning-capable MoE, it is worth checking:

• Whether your backend exposes the model's reasoning in a form Forge can capture.
• Whether an intermediate proxy or client strips reasoning fields.
• Whether reasoning_replay="full" improves your own workload.
• Whether the score improvement is worth the additional context and token cost.

You can restore replay-all behavior with:

runner = WorkflowRunner(
    workflow,
    client,
    reasoning_replay="full",
)

or through the proxy:

--reasoning-replay full

Is Forge's default changing?

Not for now.

none remains the efficient and appropriate general default based on the broader survey, particularly for smaller and dense models. Full replay carries a real context and token cost, and the smaller MoEs did not benefit.

The new results add an important model-specific caveat: users running large MoEs should test replay policy explicitly rather than assuming the default is optimal for their model.

Full replay tables are available in the published reasoning-replay results. The policy design and earlier evidence are documented in ADR-017.