A knowledge base + docs + notes for reverse engineering neural networks, with transformer language models as the core subject.
Research snapshot: 2026-08-30
Audience: researchers and expert engineers
Format: portable GitHub Markdown
Coverage: deep LLM coverage, with scoped extensions to vision, multimodal, diffusion, reinforcement learning, state-space, scientific, and agentic systems
Mechanistic interpretability (MI) studies the internal computations that produce model behavior. It identifies candidate variables and operations, tests their causal role, and states the conditions under which an explanation holds. Correlation can locate a candidate mechanism. A mechanistic claim requires additional evidence.
- Navigation hub: all maps, selectors, terminology, and reading routes.
- Field atlas: the field organized by causal unit, operation, and goal.
- Method selector: technique selection by research question and evidence requirement.
- Expert roadmaps: efficient reading routes through the atlas.
- Notation and tensor shapes: shared mathematical conventions.
- Glossary: canonical definitions and aliases.
- Evidence guide: what observational, interventional, replicated, and benchmarked evidence establishes.
- End-to-end research workflow: progression from a behavioral question to a falsifiable mechanistic investigation.
| Area | Coverage | Hub |
|---|---|---|
| Foundations | Definitions of features, circuits, mechanisms, and explanations | Foundations |
| Methods | Inspection, localization, attribution, and causal testing | Methods |
| Representations | Candidate variables, bases, and learned decompositions | Representations |
| Circuits | Composition of components into algorithms | Circuits |
| Steering and editing | Controlled behavioral changes from internal interventions | Steering and editing |
| Case studies | Detailed reverse-engineering results and their limits | Case studies |
| Evaluation | Tests that separate useful explanations from interpretability artifacts | Evaluation |
| Applications | Bounded uses in science, engineering, and safety | Applications |
| Tools | Software, models, feature sets, and compatibility guidance | Tools |
| Frontier | Unsettled research directions from 2025–2026 | Frontier |
| Extensions | Transfer beyond autoregressive language models | Extensions |
| Resources | Primary papers, documentation, curricula, and datasets | Resources |
- Specify behavior. Define a distribution, contrast, and metric. A single illustrative prompt is insufficient.
- Choose a mediator. Decide whether the candidate causal unit is a head, neuron, residual direction, learned feature, edge, path, or another variable.
- Localize. Use observation, attribution, probing, or interventions to find candidates.
- Interpret. Form a functional hypothesis that predicts activations, downstream effects, and counterfactual behavior.
- Validate. Test necessity, sufficiency, specificity, robustness, and generalization with interventions and controls.
- Compare. Measure against simple baselines and standardized benchmarks where available.
- Calibrate the claim. State exactly what was shown and what remains unexplained.
This decomposition, description, and validation loop follows the synthesis in Open Problems in Mechanistic Interpretability. The treatment of causal units follows the 2026 survey The Quest for the Right Mediator.
- Attention weights describe routing patterns. An explanation also needs value content and causal evidence.
- Decodability establishes that information is available to a classifier. It does not establish that the model uses that information.
- A successful edit identifies an effective control point. It does not uniquely localize the original computation.
- A sparse autoencoder latent is a coordinate in a learned dictionary. Monosemanticity, causal relevance, and improvement over neurons require separate tests.
- A compact circuit can preserve a chosen metric while omitting alternative or redundant mechanisms. Uniqueness requires additional evidence.
- A natural-language activation description is a hypothesis. Faithfulness requires prediction and intervention.
- Terms such as thinking, planning, and workspace are operational labels unless an experiment defines them precisely.
See Evidence and claim discipline and Failure modes.
Topic files are the source of truth. Roadmaps, indexes, and the glossary link to them instead of duplicating explanations. Every substantive page records its evidence modes and last verification date. See STYLE-GUIDE.md, CONTRIBUTING.md, and MAINTENANCE.md.
This edition treats methods and learned representations as scientific instruments with measured operating limits. Visualizations and feature labels generate hypotheses. Causal interventions, reconstruction checks, rival hypotheses, held-out predictions, and replication determine what those hypotheses support. Current preprints and laboratory reports are marked Preliminary and remain distinct from peer-reviewed or independently reproduced findings.