Transformer

Sparser, Faster, Lighter Transformer Language Models
https://github.com/SakanaAI/sparser-faster-llms

LT2: Linear-Time Looped Transformers
https://charlesdddd.github.io/lt2/

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
https://arxiv.org/abs/2604.21215

Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
https://arxiv.org/abs/2507.07814

NRGPT: An Energy-based Alternative for GPT
https://arxiv.org/abs/2512.16762

Interleaved Head Attention
https://arxiv.org/abs/2602.21371

On the Ability of Transformers to Verify Plans
https://arxiv.org/abs/2603.19954

A Mechanistic Analysis of Looped Reasoning Language Models
https://arxiv.org/abs/2604.11791

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
https://arxiv.org/abs/2603.05498

Natten - Fast Multi-dimensional Sparse Attention
https://github.com/SHI-Labs/NATTEN

Krause Synchronization Transformers
https://arxiv.org/abs/2602.11534

LUCID: Attention with Preconditioned Representations
https://arxiv.org/abs/2602.10410

Geometric and Dynamic Scaling in Deep Transformers
https://arxiv.org/abs/2601.01014

Generating Directed Graphs with Dual Attention and Asymmetric Encoding
https://arxiv.org/abs/2506.16404

Decomposing Query-Key Feature Interactions Using Contrastive Covariances
https://arxiv.org/abs/2602.04752v1

How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
https://arxiv.org/abs/2601.19208

TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors
https://arxiv.org/abs/2601.17958

In-Context Algebra
https://algebra.baulab.info/

On the Emergence of Induction Heads for In-Context Learning
https://arxiv.org/abs/2511.01033

Stronger Normalization-Free Transformers
https://github.com/zlab-princeton/Derf

Deep Delta Learning
https://github.com/yifanzhang-pro/deep-delta-learning

Attention Is Not What You Need
https://arxiv.org/abs/2512.19428

Higher-order Linear Attention
https://arxiv.org/abs/2510.27258
https://yifanzhang-pro.github.io/MEA/

Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics
https://arxiv.org/abs/2512.12602

Tensor Product Attention Is All You Need
https://arxiv.org/abs/2501.06425

Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
https://www.arxiv.org/abs/2510.01450

Stability of Transformers under Layer Normalization
https://arxiv.org/abs/2510.09904