Learn

Learn

The learning path, one mechanism per page. Each repository page gives the problem, the basic idea, what the microscope measures, what would show value, and the experiments that exist.

flowchart LR
  A[1 Dense versus MoE] --> B[2 Routing and top-k]
  B --> C[3 Sparse dispatch]
  C --> D[4 Delta experts]
  D --> E[5 Recurrence]
  E --> F[6 Engram]
  F --> G[7 Distillation]
  G --> H[8 Quantization]
  H --> I[9 Expert cache]
  I --> J[10 Heterogeneous execution]

Loading

Step	Question	Page	Status
1	How good is a tiny dense model?	Dense versus MoE	measured
2	Can we store more than we execute, and what does the second expert cost?	Routing and top-k	measured
3	Can conditional compute avoid the unused work?	Sparse dispatch	measured
4	How many experts can we afford?	Low-rank delta experts	measured
5	Can compute replace parameters?	Recurrence	planned
6	Can lookup memory replace learned memorization?	Engram	planned
7	Can a teacher improve a tiny student?	Distillation	fixture exists
8	How many experts fit in the same storage?	Quantization	projected
9	Does the whole model need to be resident?	Expert cache	estimated
10	Should every expert run on the same device?	Heterogeneous execution	planned

Index: docs/concepts/README.md, which also holds the four-sparsities table and the parts list. Background on how experts specialize, how routing decides, and how Engram is sized: docs/research/moe-engram-discussion.md.

The campus docent, a second training area whose domain is the live campus site, is documented in Experiments (CD00 onward) and in the plan.

Numbers behind each page: Results. The runs themselves: Experiments.