Preliminary work. Under review at ICML.
We introduce Functional Attention (FuncAttn), which reinterprets attention as a functional correspondence between adaptive bases rather than pairwise affinities between tokens. Inspired by the functional maps framework, FuncAttn replaces the dense softmax score matrix with a compact linear operator learned via optimal least-squares in a spectral space, reducing complexity from O(n²) to O(k²) with k ≪ n.
Qualitative comparison against Transolver on Elasticity (top) and Darcy flow (bottom). FuncAttn produces predictions with lower relative error.
| Directory | Description |
|---|---|
Few-Shot-Regression/ |
Few-shot sinusoid regression comparing FuncAttn, Attention, Intention, Transolver |
PDE-StandardBenchmark/ |
Six PDE benchmarks: Darcy, Navier-Stokes, Airfoil, Pipe, Plasticity, Elasticity |
RNA-Segmentation/ |
3D point cloud segmentation on RNA structures |
Airfoil-Design-AirfRANS/ |
OOD generalization on the AirfRANS airfoil design dataset |
Burgers-Super-Res/ |
Zero-shot super-resolution on the 1D Burgers equation |
pip install -r requirements.txt
Relative L² error (%) on six standard PDE benchmarks. FuncAttn outperforms the previous best model Transolver across all tasks.
We appreciate the following GitHub repos for their valuable code base and datasets on which we built our code: