Repository navigation
Attention Mechanism
The implementation is in the MultiHeadSelfAttention class. Both branches compute non-causal, bidirectional self-attention, appropriate for an encoder classifier.
flowchart TD
X["Encoder activations X: B x S x D"] --> Q["Linear Q projection"]
X --> K["Linear K projection"]
X --> V["Linear V projection"]
Q --> QH["Split heads: B x H x S x Dh"]
K --> KH["Split heads: B x H x S x Dh"]
V --> VH["Split heads: B x H x S x Dh"]
QH --> Choice{"--sdpa enabled?"}
KH --> Choice
VH --> Choice
Choice -->|No| Matmul["Scores = Q x K^T / sqrt(Dh)"]
Matmul --> Mask["Mask padded keys"]
Mask --> Soft["Softmax + dropout"]
Soft --> Weighted["Attention weights x V"]
Choice -->|Yes| SDPA["PyTorch scaled_dot_product_attention; non-causal"]
SDPA --> Concat["Merge heads: B x S x D"]
Weighted --> Concat
Concat --> Out["Output projection"]
- Project
Xinto Q, K and V using separatenn.Linearlayers. - Split
DintoHheads of widthDh = D/H. - Compute scores
Q @ Kᵀ / sqrt(Dh). - Mask padded keys with a very negative value, softmax on the key axis, and apply dropout.
- Multiply normalized attention weights by V, concatenate heads, and project to
D.
With --sdpa, code calls torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=keep_keys, dropout_p=...) with is_causal=False. The SDPA boolean mask has True for allowed keys; the manual implementation uses pad_mask=True for padding that must be excluded.
Each TransformerEncoderLayer is pre-LayerNorm with residual connections around attention and the feed-forward stack: Linear(D, 4D) -> GELU -> Dropout -> Linear(4D, D). This is not RMSNorm and does not use RoPE or SwiGLU; positional encoding is fixed sinusoidal.
Manual attention materializes a score matrix with shape B × H × S × S; attention memory may grow quadratically with sequence length. The actual SDPA backend and efficiency depend on PyTorch, device and input characteristics—do not assume one fixed kernel.
Home · HLD · LLD · CLI · Validation This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0.