gqadict

Grouped-query attention shrinks the value dictionary a group of heads can read by exactly the group size, and yet it costs no output rank at all: the group's realized output subspace has the same rank as full multi-head attention. The intuition that a shared value matrix halves representational rank is exactly void. The cap only reaches the output if the attention patterns are tied too, which grouped-query attention never does.

Grouped-query attention (Ainslie et al. 2023) shares one key-value pair across a group of G query heads. Each head output is o_h = P_h V with P_h = softmax(Q_h K^T / sqrt(d)) row-stochastic and V the shared N x d_v value matrix.

The value dictionary is G-fold smaller

The set of value vectors any head in the group can produce is the span of the outputs of a row-stochastic pattern against V, which is exactly the row space of V, of dimension min(N, d_v). Full multi-head attention gives each head its own V_h, so the group's value dictionary is min(N, G d_v). Sharing the value matrix reduces the dictionary by exactly a factor of G (measured 16 = d_v for GQA versus 128 = G d_v for MHA at G = 8): all G heads must draw their outputs from one set of N value vectors rather than G independent sets.

It costs no output rank

The layer output stacks the residual writes, C = [P_1 V W_O^1 | ... | P_G V W_O^G], each head with its own output projection W_O^h. Even though every block reuses the one shared dictionary, each is post-composed with an independent full-rank pattern P_h and an independent W_O^h, so the blocks do not couple and

rank(C) = min(N, G d_v)   generically,

exactly the rank of full multi-head attention with independent per-head values (measured GQA 128 = MHA 128 = min(N, G d_v), and equal across every swept configuration). The free attention patterns re-diversify the single d_v-dimensional dictionary into G distinct d_v-dimensional images that tile the full G d_v feature space. Sharing V costs cache, not output rank.

The cap bites only if the patterns are tied

If the whole group shared one attention pattern P, then C = P V [W_O^1 | ... | W_O^G] and rank(C) <= d_v (measured 16 = d_v): the value-dictionary cap would reach the output. It is pattern freedom, not value freedom, that carries the output rank. Grouped-query and multi-query attention keep an own Q_h per head, so the patterns stay free and the cap never triggers.

Layout

• dictionary.py: the value-dictionary rank, d_v for GQA versus G d_v for MHA.
• output.py: the output-concat rank equality min(N, G d_v) for GQA and MHA, and the tied-pattern collapse to d_v.
• test_gqadict.py: the G-fold dictionary cap, the output-rank equality, the cap-bites-iff-patterns-tied condition, and the equality across configurations.

Reproduce

python dictionary.py
python output.py
python test_gqadict.py

Note

The linear algebra here, the row space of a shared matrix and the rank of a block concatenation, is elementary, and this repository claims none of it. The grouped-query and multi-query papers motivate key-value sharing for cache size and report empirical quality; neither states the group-level value-dictionary rank law or the pattern-freedom cancellation. The single-head fact that one head's output lands in a d_v-dimensional subspace is known; the group-level question, whether sharing V across G heads lowers the joint output rank, and its exact answer of no with the exact iff-condition, is what is here. It is distinct from the shipped attnrank (the rank of the attention matrix, which is used here as the mechanism that keeps the patterns full-rank) and ropelowrank (key rank under RoPE). The named extension is the measured effective rank (stable rank or participation ratio) of the group output on trained grouped-query checkpoints against the min(N, G d_v) ceiling and an MHA control, to see how much of the ceiling real models with correlated output projections actually use.