SpecFormer: Spectral-Aware Transformer for Recommendation Systems

A spectral-aware Transformer architecture that incorporates SVD-based spectral information into multi-head attention for better feature interaction modeling in CTR prediction tasks.

Architecture

SpecFormer extends standard Transformer MHA with three key spectral components:

(1) Learnable Spectral Softening

SVD decomposition on input token matrix:

H = U Σ V^T

Singular value softening:

Σ* = diag(σ_1^τ, σ_2^τ, ..., σ_r^τ)
τ = Sigmoid(a), a is learnable, initialized to 0

(2) Spectral-Aware Feature Interaction Attention

Q, K from spectral-softened H*:

H* = U Σ* V^T
Q = H* W_Q,  K = H* W_K
S = Q K^T / sqrt(d)

Value matrix: V = H (no softening, no projection)

(3) Learnable Spectral-Aware Residual Connection

Polynomial spectral bias matrix:

A_bias = U M U^T,  M_ij = σ_1² · f(σ̄_i) · f(σ̄_j)
σ̄_i = σ_i / σ_1
f(σ̄) = β₀ + β₁·σ̄ + β₂·σ̄²   (learnable coefficients)

Vanilla attention residual:

S_bias = H W_Q' (H W_K')^T / sqrt(d)

Full attention:

SpecAtt(H) = softmax(S + α·A_bias + β·S_bias) · V

Residual + LayerNorm + PFFN:

Z = LN(SpecAtt(H) + H)
H' = LN(Z + PFFN(Z))

Quick Start

from models.spectral_mha import SpectralMHAModule

config = {
    'enable_spec_qk_from_hstar': True,
    'enable_sigma_soft_power': True,
    'enable_sigma_soft_learnable_tau': True,
    'enable_spec_pos_enc_poly': True,
    'enable_vanilla_fusion': True,
    'disable_v_proj': True,
}

module = SpectralMHAModule(config)
output = module(inputs, num_heads=8, use_spectral_bias=True)

See models/spectral_mha.py for dataset-specific optimal configurations and full config options.

Directory Structure

Specformer/
├── __init__.py
├── README.md
├── models/
│   ├── __init__.py
│   └── spectral_mha.py          # Core SpectralMHA module + config reference
└── ops/
│   ├── __init__.py
│   └── spec_ops.py               # SVD purification, sigma softening, spectral bias

Dependencies

• TensorFlow >= 1.14

Citation

@article{specformer2026,
  title={SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
}