Start Here

Start Here

What it is

MicroMoE is a transformer of a few thousand parameters trained on a synthetic domain of four task families (arithmetic, sequences, MLPL fragments, prose) over a 52-symbol alphabet. Small enough that the whole model fits on one screen; real enough that routing, specialization, and sparse dispatch behave the way the large-scale papers describe.

Why

FreeToken-style inference wants a model that stores far more than it executes: experts on disk, hot experts in a cache, memorized patterns in a lookup table, depth from recurrence rather than layers. Each of those is a kind of sparsity, and each can be shown separately at microscope scale.

flowchart TB
  P[Parameter sparsity: recurrence] --> M[MicroMoE]
  C[Compute sparsity: top-k routing and sparse dispatch] --> M
  L[Memory lookup sparsity: Engram] --> M
  R[Residency sparsity: expert cache] --> M

Loading

What we have learned so far

• Top-1 routing stores 1.86 times the parameters of the dense model at 1.02 times the active parameters per token, at unchanged validation loss.
• Experts specialize by task family without seeing a family label (specialization 0.61 against a 0.25 blind baseline).
• Top-2 fits better, costs twice the expert work, and flattens the map.
• Data moves the held-out columns; more epochs at the same data do not.
• Sparse dispatch is exact and does a quarter of the expert work, and is still slower in the interpreter.
• Reusing one block three times buys quality per parameter (validation loss 3.42 at the mixture's 7,096 parameters), and the router re-decides at every recurrence; it is not a converging reasoning loop.
• An Engram table written from scratch equals the sw-MLPL builtin to the bit; at 90 training windows it memorizes instead of generalizing, alone and composed with routing and recurrence.
• A trained docent does not yet beat keyword matching on paraphrases.

Each claim, with its evidence and limitation, is in docs/results/current-findings.md.

Next

Read the mechanisms in order on Learn, or jump to Results.