A few new models on the dashboard — small MoE, 16GB tier (and what's next)

[antoinezambelli]

Folded three small-MoE models into the gen-3 eval set this week — Mellum2-12B-A2.5B (Thinking + Instruct) and LFM2.5-8B-A1B, all in the 16GB tier (high total params, low active — the kind that run comfortably on modest local hardware). n=50, native + prompt, reforged + bare, same suite and dataset as everything else. Not a release — gen is a comparability epoch, not a version, so these slot in as same-gen peers with no version bump.

The one that stands out: Mellum2-12B Instruct, reforged native, scores 81.0% — and does it at 1.2s/call. That lands it right in the top Ministral-3 pack (the local bracket tops out around 84.5%), but ~3–7× faster than anything sitting near it. For agentic loops where latency compounds over many turns, that's the interesting number, not the headline accuracy.

The honest texture

• The Thinking variant underperformed Instruct (68.4% vs 81.0% reforged native) and ran 7–9× slower for the privilege. Counterintuitive, and a reminder that "thinking on" isn't a free win on this kind of bounded, protocol-heavy task.
• LFM2.5-8B-A1B struggled — 57.7% reforged native, worse in prompt mode. A1B = 1B active params; this suite leans hard on instruction-following discipline and the smallest active-param model in the set shows it.
• Replay coverage is none-only for all three — not the full none/keep-last/full grid the other gen-3 8–14B models carry. A re-sweep fills that in.

Same forge-lift story underneath: these are bare-model vs. the guardrail layer, nothing more. Dashboard's refreshed — full tables, per-scenario breakdowns, and the carried-forward older generations are all in docs/results/.

What's next: auth

The next big work item is auth, landing in 0.8.0 — the broader-version slice from the roadmap. forge today assumes a local unauthenticated backend (or the Anthropic SDK's own path); 0.8.0 brings auth to the proxy layer and forge generally, so whatever client you point at it, with whatever credentials the backend wants, it just works — forge forwards what it's given and stays out of the way. There are a couple of community PRs already circling this surface (#104, #106, thanks for the patience there); 0.8.0 is where it gets a unified design rather than per-backend one-offs.

Links

• Dashboard / full tables: docs/results/
• EVAL_GUIDE (incl. eval generations): docs/EVAL_GUIDE.md

---

PS — if anyone has llama.cpp settings or sampling params that pull better numbers out of LFM2.5-8B-A1B, I'd genuinely like to see them. Felt like there was more in there than 57.7%.