Value Embeddings idea

[josh-gree]

The paper (Zhou et al., "Value Residual Learning") is about reusing the first layer's value projections in later layers - a skip connection over the residual stream. That's meaningfully different from what ended up in the code: a learned embedding table indexed by token ID, injected into attention values at each layer.

Where did the embedding idea come from? A paper, a tweet, a Discord discussion? I think directly it probably came from the modded-nanogpt repo? But not really having much luck in understanding how it came to be there?

[Mithun-033]

Was wondering the same.
Would love some insight @karpathy

[jenna-russell]

Disclaimer this is GPT Sol's answer, but I was also curious about the reasoning:

The lineage appears to be:
Zhou et al., “Value Residual Learning” (https://arxiv.org/abs/2410.17897, submitted October 2024). Their core idea is to preserve an early layer’s attention values and mix them into later layers:
V_l = λ_current · V_l + λ_early · V_early
The motivation is that deep attention layers dilute token-level information.

modded-nanogpt adapted this into token-indexed lookup tables. The December 4, 2024 record by KoszarskyB added separate nn.Embedding tables, looked up by token ID, and mixed those vectors directly into attention’s V stream. The code explicitly comments that this was inspired by Grad62304977’s value-residual implementation. https://github.com/KellerJordan/modded-nanogpt/blob/master/records/track_1_short/2024-12-04_ValueEmbed/train_gpt2.py#L1644-L1668.

In other words:
paper: V_l + earlier computed value state
nanochat: V_l + learned embedding[token_id]

nanochat inherited the modded-nanogpt version. Karpathy’s discussion says the winning ideas—including value embeddings—originated in modded-nanogpt.

[josh-gree]

Thanks for this - it's really how it got to be in modded nano chat that I was particularly interested in...