Universality of Empirical Risk Minimization

Numerical experiments exploring the "universality" of empirical risk minimization (ERM) in high dimensions.

Recent work in high-dimensional statisitcs reveals a phenomonon where complex non-linear models have equivalent training and generalization error to corresponding linear gaussian models in the high-dimensional asymptotics $n,p \to \infty, \ \frac{p}{n} \to \gamma \in (0, \infty)$. Here, we explore whether this universality phenomenon holds in settings more general than the ones already studied.

Relevant papers:

Universality Laws for High-Dimensional Learning with Random Features --- Hong Hu & Yue M. Lu (2022)

Universality of Empirical Risk Minimization --- Andrea Montanari & Basil Saeed (2022)