-
Notifications
You must be signed in to change notification settings - Fork 0
Home
A tiny mixture-of-experts language model with an Engram memory table, built from scratch in sw-MLPL, that you can see all of. Every step, data structure, and transformation has an annotated diagram; every lesson is measured for memory, speed, and quality.
Live demo: https://sw-ml-study.github.io/moe-microscope/, the pinned recordings stepped frame by frame, and the panned animation https://sw-ml-study.github.io/moe-microscope/landscape.html, the same recordings as data moving across one wide scene (both playbacks; no sw-MLPL code runs in the browser). Blog post: https://blog.softwarewrighter.com/2026/09/13/saw-building-a-tiny-mixture-of-experts/.
This wiki is a navigation layer. The repository documents are canonical: docs/README.md is the landing page and README.md the executive summary.
flowchart LR
A[Dense model] --> B[Add experts]
B --> C[Route to top-k]
C --> D[Skip inactive experts]
D --> E[Reuse one block recursively]
E --> F[Add Engram memory]
F --> G[Quantize experts]
G --> H[Keep hot experts resident]
H --> I[Schedule across CPU and NPU]
| If you want to | Start at |
|---|---|
| Understand the idea in ten minutes | Start Here |
| Learn the mechanisms one at a time | Learn |
| Know whether it works, with numbers | Results |
| Read one laboratory report | Experiments |
| Modify or reproduce it | Internals |
| Look something up | Reference |
Sagas 1 to 3 are complete (foundation, MoE from scratch, resource budget and documentation), and so is Saga 5 (recurrence and Engram: one block reused with deep supervision, routing over reasoning time, the Engram table from scratch with exact parity against the sw-MLPL builtin, and the seven-row ablation matrix of all three sparsities). Saga 4, the campus docent, is paused after its first five steps (snapshot, dense docent, live demo page, matcher baseline, word vectors): the docent does not yet beat a deterministic matcher on paraphrases and joins the campus site only when it does. Next: an animated data-flow landscape played from the pinned recordings, then a Mamba-style state-space block and the tiny hybrid, distillation, the state-memory-experts composition, multi-token prediction, the routed docent, quantization, expert caching, heterogeneous execution, the interactive host, the configuration frontier, the CUDA move, and the final findings report.