This repository serves as a note-keeping library for papers related to applying SAE in hopes of achieving mechanistic insights into the life2vec model, hopefully uncovering interesting and interpretable features.
The following list is not fully updated (25-07-2025) and may very stay as such. I refer you to the thesis references for more in-depth review of relevant material.
The following papers are the primary ones contributing to the thesis and its foundational principles:
| Status | Paper Title (Year) | Description | Link |
|---|---|---|---|
| ✔️ | Anthropic: Towards Monosemanticity (2023) | Uses L1 and L2 loss to promote sparsity in SAE. Good plots. Some critiques? Parallel to Cunningham et al., based on this partially. (More notes required) | Link |
| ✔️ | Anthropic: Scaling Monosemanticity (2024 follow-up) | Follow-up of above, focusing on the scaling of SAE's to the larger LLM's. | Link |
| ✔️ | Life2Vec (Nature): Using sequences of life-events to predict human lives (2023) First read-through (65%) |
Represents life-events as a sequence and trains on registry data using a BERT model. More models exist now (HGMAE, GPT-like soon™️), SOTA baseline. | Link |
| ✔️ | BERT – Bidirectional Encoder Representation of Transformers (Google, 2018) | Contextualizes attention bidirectionally through masking. Transformer structure. | Link |
Materials related to Mechanistic Interpretability using Sparse Autoencoders on LLMs/Transformers. The checklist helps keeping track of progress (📖 indicates “currently reading”). A brief description to capture the primary insights from each resource is available.
| Status | Resource Title (Year/Info) | Description | Link |
|---|---|---|---|
| ✔️ | Anthropic: Update on SAE training regime (2024) | (Key points summary…) | Link |
| ✔️ | Anthropic: Toy Models of Superposition (2022) | Investigates superposition principle in Transformers using small toy models. Very informative w.r.t. bases and relates back to several papers below. | Link |
| ✔️* | Sparse Crosscoders for Cross-Layer Features and Model Diffing (2024) | (Key points summary…) | Link |
| ✔️ | Chris Olah – Dark Matter of AI – MechInterp (2024) | (Key points summary…) | Link |
| ✔️ | Privileged Bases in Transformer Residual Stream (2023) | Identifies ADAM as possible culprit of privileged bases - they observe heavy-tailed activations inside the computation basis, but not in the residual stream. | Link |
| ✔️ | Chris Olah – Zoom In: An Introduction to Circuits (2020) | First notion of circuits. Visual explanation based on InceptionV1 (iirc), good grokking paper. | Link |
| ✔️ | A Mathematical Framework for Transformer Circuits (2021) | Foundational paper describing a Transformer as a circuit (QK, OV) and a mathematical way of understanding said circuits. Good entry for understanding foundational principles. | Link |
| ✔️ | The Geometry of Concepts: Sparse Autoencoder Feature Structure (2024) | (Key points summary…) | Link |
| ✔️ | Scaling and Evaluating SAE's (2024) | (Key points summary…) | Link |
| ✔️ | Neel Nanda – MechInterp (SAE) podcast (2024) | General rant about open problems and interesting avenues/ways to think about mechanistic interpretability in a fast and growing field of research. | Link |
| ✔️ | DeepMind – Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024) | (Key points summary…) | Link |
| ❎ | Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions (2024, Magnus) | (Key points summary…) | Link |
| ✔️ | SAE's find highly interpretable features in LLM's (2023) | Uses autointerpretability (Bills et al.) with an SAE to intervene on activations layers in Pythia model. Comparison baselines are (1) default (2) random directions (3) PCA (4) ICA. Later layers less interpretable? Uses activation patching and ACDC for causality checks. | Link |
| ❎ | (BERT?) Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors (2023) | (Key points summary…) | Link |
| ❎ | Codebook features: Sparse and Discrete interpretability for neural networks (2023) | (Key points summary…) | Link |
| ✔️ | YouTube: 3Blue1Brown – Visual explanation of LLM and Transformers (series) | (Key points summary…) | Link |
| ✔️ | YouTube: Welch Labs – Dark matter of AI | (Key points summary…) | Link |
| ✔️* | OpenAI: LLM can explain neurons in LLM (2023) | Pretty sure this is AutoInterp or early version thereof. Probably not useful with our data. Explains correlations, not mechanisms. | Link |
| ✔️ | Word2Vec – Efficient Estimation of Word Representation in Vector Space (2013) | (Key points summary…) | Link |
| ✔️ | Google DeepMind – Improving Dictionary Learning with Gated SAE's (2024) | Combats bias in loss from L1 by harming L2. There's some valuable comments about the feature shrinkage phenomenon, which Gated combats here. Neel's team's preferred way of training SAEs. Makes a "gate", s.t. JumpReLU behaviour on preactivation. Numerous other important definitions!! | Link |
| ❎ | Improving SAE's by SQRT-L1 and Removing lowest activating features (article, 2024) | (Key points summary…) | Link |
| ❎ | Scratchpads: Show your work – Intermediate computation with LLM's (used in Anthropic) (2021) | (Key points summary…) | Link |
| ✔️ | Neel Nhanda – An opinionated list of favourite papers v2 (2024) | Contains primary papers for new people in the field and highlights important sections or ideas of each paper to focus on. | Link |
| ✔️ | GitHub with MechInterp in LLM's – Code and paper references | A general repo I found. | Link |
| ✔️ | Decoding the Thought Vector (early work on sparsity) (2016) | Early work on sparsity and to some extent superposition (iirc). Primarily through the lens of visual model. | Link |
| ✔️ | Multimodal neurons in ANN (fires regardless of modality) (2021) | Despite modality a concept can be recognized in neuron activations. | Link |
| ✔️ | Comments and counter-points to Anthropic's "Scaling" paper | Unclear if dictionary learning can find specific features of interest. Correlations between neurons and features seems high, despite Anthropic's claim? Mostly on limitations or claims made that might be deceiving. | Link |
| ✔️ | Interpretability Illusion for BERT (2021) | Strong evidence for superposition and illusions of concepts across datasets (intuitively makes sense imo). BERT activation space orgnanizes similar sentences in clusters, but linear directions in activation space do not correspond to linguistic concepts. Early work. | Link |
| ✔️ | (Return to this, original SAE idea?!) [Interim research report] Taking features out of superposition with sparse autoencoders (2022) | Highlights some challenges in terms of disentangling the complex neuron representations from the SAE. Uses MMCS, finds Goldilocks zone for L1 and dictionary size, important to track dead neurons and loss "stickiness" to identify optimal hyperparameters. (see Anthropic for more nuance?) | Link |
| ✔️ | Grammar of Life – Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages (2025) | Features capture generalizable abstractions in language. Uses Gated SAEs to reduce bias from L1 term (unlike Anthropic), which "separates selecting features to use and estimating activation magnitude of those directions ". Attribution patching for causally relevant features. Quality of SAEs by measuring proportion of loss recovered when replacing model activations with SAE reconstructions thereof. | Link |
| ✔️ | Tom McGrath - SAE takes w.r.t. AutoInterp (Robert Huben's pointer). | Auto-Interp has its limitations, such as scoring really well on "trivial features", but can be a useful tool in attempting to interpret each feature. | Link |
| ✔️ | [Research Report] SAE find only 9/180 (33/180) board state features in OthelloGPT | SAE might not find a significant number of features from the LM, in this case corresponding to human-understandable ontology of these features (meaning we can categorize and label it). So, they show despite having features we understand, the SAE might not uncover all of them. | Link |
| ✔️ | Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders | Has a positive threshold which indicates when a feature is considered active (and separates it from its magnitude). Alternative to Gated SAE's (Aug vs. Apr), similar interpretability but slightly better reconstruction (at fixed sparsity). Bypasses L0-like loss (L1) and directly uses L0. Estimated via straight-through estimators (STE). Uses KDE to estimate expected probabilities w.r.t. L0 gradient, and they use pseudo-derivatives. Has higher frequency features (like TopK) compared to Gated SAE's, but considered mild improvement of prevailing methods due to no auxillary term. | Link |
| ✔️ | Sparse Autoencoders Can Interpret Randomly Initialized Transformers | They use auto-interp on randomly initialized transformers - the parameters are sampled IID from a Gaussian rather than trained on text data - and find their SAE latents are similarly interpretable as trained variants compared to a control model across different sizes and layers. They conclude (under this metric) this implies the SAE does not capture the computational structure of the model, but rather are inherent to the sparsity of the text data its trained on. | Link |
| ✔️ | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | Concludes that while sparse autoencoder (SAE) probes sometimes show isolated improvements, they do not consistently outperform traditional baseline methods across various challenging regimes (data scarcity, class imbalance, label noise, and covariate shift) on 110+ datasets. In essence, the claimed benefits of SAE probes appear illusory when rigorously benchmarked against strong baselines, underscoring the need for more robust evaluation practices in mechanistic interpretability research. | Link |
| ✔️ | Tracing the Thoughts of a Large Language Model | Blog post with links to the two papers by Anthropic here and here | Link |
| ✔️ | Dario Amodei - Urgency of Interpretability (Blog post) | CEO of Anthropic with thoughts (politicized). LessWrong post here | Link |
| ✔️ | Negative Results for SAEs on Downstream Tasks and Deprioritising SAE Research (GoogleDeepMind MechInterp Team Update #2) | SAE's are useful but field is possibly overly invested in them (Anthropic). Tried usefulness on downstream task of OOD generalization for harmful intent in user prompts. Underperformed compared to linear probes. | Link |
| ✔️ | Interpretability Will Not Reliably Find Deceptive AI (Neel Nanda blog) | *His views reflect that interp is a valuable tool, but will only be one layer in our defence capacity for evaluating or monitoring the safety of super-intelligent systems, given current research paradigm. He disagrees with Dario Amodei's recent blog post that it is the only way forward. * | Link |
| ✔️ | LLaMA Scope: Extracting millions of features from LLama-3.1-8B with Sparse Autoencoders | META's equivalent to Gemma Scope above. Has some nice pointers to sizes and method's, e.g. 2D UMAP of decoder columns and difference between an encoder and decoder view of features. | Link |
| ✔️ | Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations | (Key points summary...) | Link |
| ✔️ | Chain-of-Thought Reasoning In The Wild Is Not Always Faithful | (Key points summary...) | Link |
| ✔️ | Sparse Autoencoders Do Not Find Canonical Units of Analysis | (Key points summary...) | Link |
| ❎ | Alignment faking in large language models | Demonstration of LLM's selectively complying with its training objective in training to prevent modification of its behavior out of training. Future risks etc. | Link |
- k-top SAE, Gated SAE, JumpReLU (per Neel’s suggestions Gated is the way to go? See comments on papers. (Pros/Cons: Resampling, reconstruction vs sparsity, L0-like (L1) or L0 (pseudo-derivative))
- Gated SAE's are better than what Anthropic did, because they avoid the bias introduced from L1 w.r.t. harming L2. (Rajamanoharan et al. (2024))
- JumpReLU are easier and better? I've heard different things different places...
- GeLU vs. ReLU: Which activation function works best in practice? (Neel says GeLU...)
- Measuring SAE performance: What are the most effective evaluation metrics? (Huben email sources, his input is to focus on how you can evaluate the features)
- [Huben] Check if your features serve as a classifier for an external set of interpretable concepts, see his work, Karvonen et al., and Gao et al., section 4.2
- [Huben] Find a way to represent your features so that their usefulness is immediately visible, see e.g. his attempt or Anthropic's graphs.
- [Toy Model Comment] States Linear-Algebra way to deem leverage or "efficiently packing" the dimensionality of all features s.t. empirically they add up to the number of embedding dimensions - he explains why this is natural w.r.t. rank/covariance/lemma from numerical algebra
- Evidently, AutoInterp is a problem unto itself, but it cannot be leveraged in this project given the structure of the data, the sensitive nature of its contents, and the code-token mapping being OOD for standard LLM's.
- GitHub of SAE training code
- ARENA course and exercises (recommended by Neel) Although this one only uses modular libraries e.g. TransformerLens and SAELens, so it is not with your own models!!! Super nice for learning though 😄