Why is attention quadratic and does it actually matter at my scale? #29
Verified answer
by
martex-dev
martex-dev
asked this question in
Q&A
|
Why is attention quadratic and does it actually matter at my scale? Notes and answer below. |
Admin verified this answer by
martex-dev
Aug 27, 2026
Replies: 1 comment
|
Self-attention compares every token to every other token, so time and memory both scale with the square of sequence length. Below a few thousand tokens it rarely dominates the budget. It starts to matter when the attention matrix no longer fits comfortably in memory, which is why FlashAttention-style tiling, which never materialises the full matrix, gives a large win at long context even though the asymptotic cost is unchanged. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Self-attention compares every token to every other token, so time and memory both scale with the square of sequence length. Below a few thousand tokens it rarely dominates the budget. It starts to matter when the attention matrix no longer fits comfortably in memory, which is why FlashAttention-style tiling, which never materialises the full matrix, gives a large win at long context even though the asymptotic cost is unchanged.