This repo is a companion to the paper
A Fine-Grained Spectral Perspective on Neural Networks
Greg Yang and Hadi Salman
It serves several purposes
- It overviews the results of the paper in two self-contained jupyter noteboks (see Neural Network Generalization and Clarifying Simplicity Bias). For most of the potential users of this repo, we recommend reading one of them to start. They have overlapping content in the part of overviewing the paper's theoretical results.
- It contains the computational tools (
theory.py) contributed by the paper for computing eigenvalues of neural tangent kernel and conjugate kernel (or NNGP kernel) over the boolean cube, and also examples for how to use them (see any of the notebooks). - It contains a dump of plots describing how the different hyperparameters such as depth and initialization scheme conspire to change the kernels (see The Complete Hyperparameter Picture).
