Skip to content

Attention Mechanism

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Multi-head Self-Attention — Internal Design

The implementation is in the MultiHeadSelfAttention class. Both branches compute non-causal, bidirectional self-attention, appropriate for an encoder classifier.

flowchart TD
    X["Encoder activations X: B x S x D"] --> Q["Linear Q projection"]
    X --> K["Linear K projection"]
    X --> V["Linear V projection"]
    Q --> QH["Split heads: B x H x S x Dh"]
    K --> KH["Split heads: B x H x S x Dh"]
    V --> VH["Split heads: B x H x S x Dh"]
    QH --> Choice{"--sdpa enabled?"}
    KH --> Choice
    VH --> Choice
    Choice -->|No| Matmul["Scores = Q x K^T / sqrt(Dh)"]
    Matmul --> Mask["Mask padded keys"]
    Mask --> Soft["Softmax + dropout"]
    Soft --> Weighted["Attention weights x V"]
    Choice -->|Yes| SDPA["PyTorch scaled_dot_product_attention; non-causal"]
    SDPA --> Concat["Merge heads: B x S x D"]
    Weighted --> Concat
    Concat --> Out["Output projection"]
Loading

Manual path

  1. Project X into Q, K and V using separate nn.Linear layers.
  2. Split D into H heads of width Dh = D/H.
  3. Compute scores Q @ Kᵀ / sqrt(Dh).
  4. Mask padded keys with a very negative value, softmax on the key axis, and apply dropout.
  5. Multiply normalized attention weights by V, concatenate heads, and project to D.

SDPA path

With --sdpa, code calls torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=keep_keys, dropout_p=...) with is_causal=False. The SDPA boolean mask has True for allowed keys; the manual implementation uses pad_mask=True for padding that must be excluded.

Normalization and feed-forward network

Each TransformerEncoderLayer is pre-LayerNorm with residual connections around attention and the feed-forward stack: Linear(D, 4D) -> GELU -> Dropout -> Linear(4D, D). This is not RMSNorm and does not use RoPE or SwiGLU; positional encoding is fixed sinusoidal.

Complexity caveat

Manual attention materializes a score matrix with shape B × H × S × S; attention memory may grow quadratically with sequence length. The actual SDPA backend and efficiency depend on PyTorch, device and input characteristics—do not assume one fixed kernel.

Clone this wiki locally