Skip to content

MLI-lab/early_stopping_double_descent

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Early_stopping_double_descent

This repository contains the code for reproducing figures and results in the paper ``Early Stopping in Deep Networks: Double Descent and How to Eliminate it''.

Requirements

The following Python libraries are required to run the code in this repository:

numpy
jupyter
torch
torchvision

and can be installed with pip install -r requirements.txt.

Usage

All the figures in the paper can be reproduced by running the respective notebooks as indicated below:

Figure 2: Bias-variance trade-off curves for the linear regression model can be reproduced by running the sum_bias_variance_tradeoffs notebook.

Figure 3: Double descent in the two layer neural network and the elimination of the double descent through the scaling of the stepsizes of the two layers can be reproduced by running the early_stopping_two-layer-nn_double_descent.ipynb notebook.

Figure 1-a, 4: Double descent in the 5-layer convolutional network and the elimination of the double descent through the scaling of the stepsizes of the different layers can be reproduced by running the early_stopping_deep_double_descent.ipynb notebook.

The numerical results can be reproduced by training the 5-layer convolutional network with python3 train.py --config $CONFIG_FILE where CONFIG_FILE points to the config.json file of the desired setup in the ./results/ directory.

Disclaimers

Figure 1-a, 7: The bias and variance is measured as proposed in Yang et al. [2020] but adopted to measure bias-variance at each epoch. This may result in highly noisy measurements for the early training phase (see this notebook for details).

Citation

@article{heckel_yilmaz_2020,
    author    = {Reinhard Heckel and Fatih Furkan Yilmaz},
    title     = {Early Stopping in Deep Networks: Double Descent and How to Eliminate it},
    journal   = {arXiv:2007.10099},
    year      = {2020}
}

Licence

All files are provided under the terms of the Apache License, Version 2.0.

About

Code for reproducing figures and results in the paper ``Early stopping in deep networks: Double descent and how to eliminate it''

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published