Dead Latents

I have a bunch of dead latents. I'm training width 16K SAEs on DINOv3 ViT-L activations on FishVista (very homogenous images). I am comparing TopK and ReLU + L1 sparsity penalties and I realized that many, many of my latents are dead.

Here are graphs of the number of dead latents during training, grouped by the relative sparsity penalty:

ReLU + L1 sparsity penalty dead units throughout training on FishVista.

TopK dead units throughout training on FishVista.

My options, based on prior work, include:

1. Better W_dec/enc initialization (datapoint initialization, see https://transformer-circuits.pub/2025/october-update/index.html#data-point-init)
2. [maybe] Better b_enc initialization (according to Anthropic, see https://transformer-circuits.pub/2025/january-update/index.html#DL )
3. [maybe] Gao et al's 'AuxK' loss
4. [maybe] Anthropic's pre-act loss (again, see https://transformer-circuits.pub/2025/january-update/index.html#DL)

Starting with (1) seems super easy and reasonable. Doing (2) seems like I need to do a bunch of forward passes, which seems bad. 3 and 4 seem reasonable as well, but I don't really like throwing additional loss terms into my objective.

So I am going to:

1. Train ReLU and TopK on ADE20K
2. Add better W_dec/W_enc initalization using datapoint init + transposes
3. Train ReLU and TopK on ADE20K and FishVista with the better initialization.

For all checkpoints trained on layer 24/24:

Activation	Initialization	Dataset	Dead Unit %
ReLU	Kaiming	ImageNet-1K	38.89
ReLU	Kaiming	FishVista	46.96
ReLU	Datapoint	ImageNet-1K	31.44
ReLU	Datapoint	FishVista	54.28
TopK	Kaiming	ImageNet-1K	53.08
TopK	Kaiming	FishVista	79.09
TopK	Datapoint	ImageNet-1K	47.22
TopK	Datapoint	FishVista	77.64