Improving generalization by controlling label-noise information in neural network weights

The author implementation of LIMIT method described in the paper "Improving generalization by controlling label-noise information in neural network weights" by Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan.

If you goal is to reproduce the results, please use the version of the code at time of ICML 2020 camera-ready submission. It can be found in the following release.

If your goal is to use LIMIT, you can use the this newer code. It is better commented and easier to use. The main method of the paper, LIMIT, is coded in the LIMIT class of the file methods/limit.py.

To cite the paper please use the following BibTeX:

@incollection{harutyunyan2020improving,
 author = {Harutyunyan, Hrayr and Reing, Kyle and Ver Steeg, Greg and Galstyan, Aram},
 booktitle = {Proceedings of Machine Learning and Systems 2020},
 pages = {5172--5182},
 title = {Improving generalization by controlling label-noise information in neural network weights},
 year = {2020}
}

Requirements:

• Basic data science libraries: numpy, scipy, tqdm, matplotlib, seaborn, pandas, scikit-learn.
• We use Pytorch 1.4.0, but higher versions should work too.
• Additionally, only for extracting data from Tensorboard logs, tensorflow >= 2.0 is needed.

The exact versions of libraries we used are listed in the requirements.txt file.

Using the code

The whole code is writen as a package. All scripts should be initiated from the root directory. An example command would be:

python -um scripts.train_classifier -d cuda -c configs/4layer-cnn-mnist.json --log_dir logs/mnist

UPDATE

for a plain run:

python -um scripts.train_classifier --device=cpu --config=configs/minimal-mlp-mnist.json --dataset=mnist --loss_function=ce --log_dir=logs/mnist --data_dir=data

for random labels on training, but plain validation labels:

python -um scripts.python -um scripts.train_classifier --device=cpu --config=configs/minimal-mlp-mnist.json --loss_function=ce --log_dir=logs/mnist --data_dir=data --dataset=uniform-noise-mnist --error_prob=.9 --clean_validation --epochs=50000

To monitor the training we run Tensorboard:

tensorboard --logdir=/path/to/the/log/directory

UPDATE

tensorboard --logdir=logs

Structure of the repository

Directory	Purpose
methods	Contains implementations of classifiers used in the paper, including LIMIT.
modules	Contains code that is modular and can be shared across different models and experiments.
notebooks	Designed for Jupyter notebooks. Contains the notebooks used to generate the plots in the paper.
scripts	Contains the scripts for training, testing, collecting results, and generating training commands.
configs	Stores training/architecture configurations of our models.
logs	Used to store tensorboard logs.
data	Used to store data files.
plots	Used to store the plots.
nnlib	Points to a submodule which contains useful and generic code for training neural networks.

Reproducing the results

As mentioned above, we recommend using the code of the v0.1 release to reproduce the results of the paper. However, one should be able to reproduce the results with the current code too, although some unexpected errors might arise (they should be easy to fix). The commands we used to run the experiments were generated using the scripts/generate_commands.py script.