Home

MoE Microscope

A tiny mixture-of-experts language model with an Engram memory table, built from scratch in sw-MLPL, that you can see all of. Every step, data structure, and transformation has an annotated diagram; every lesson is measured for memory, speed, and quality.

Live demo: https://sw-ml-study.github.io/moe-microscope/, the pinned recordings stepped frame by frame (a playback; no sw-MLPL code runs in the browser). Blog post: https://blog.softwarewrighter.com/2026/09/13/saw-building-a-tiny-mixture-of-experts/.

This wiki is a navigation layer. The repository documents are canonical: docs/README.md is the landing page and README.md the executive summary.

flowchart LR
  A[Dense model] --> B[Add experts]
  B --> C[Route to top-k]
  C --> D[Skip inactive experts]
  D --> E[Reuse one block recursively]
  E --> F[Add Engram memory]
  F --> G[Quantize experts]
  G --> H[Keep hot experts resident]
  H --> I[Schedule across CPU and NPU]

Loading

Where to go

If you want to	Start at
Understand the idea in ten minutes	Start Here
Learn the mechanisms one at a time	Learn
Know whether it works, with numbers	Results
Read one laboratory report	Experiments
Modify or reproduce it	Internals
Look something up	Reference

Status

Sagas 1 to 3 are complete (foundation, MoE from scratch, resource budget and documentation). Saga 4, the campus docent, is under way: the campus catalog snapshot and corpus (CD00), the first trained docent (CD01, a dense classifier trained in batch), and the live demo page are done; the matcher baseline, word vectors, the routed docent, export, browser inference, and the campus handoff follow. The docent joins the campus site only if it measurably beats a deterministic matcher. Then recurrence, Engram, distillation, quantization, expert caching, heterogeneous execution, the interactive host, the configuration frontier, the CUDA move, and the final findings report.