Reproducing the paper "PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks" for the ICLR 2019 Reproducibility Challenge
Clone or download
Latest commit e383342 Jan 14, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
resnet-18 Add plots for CIFAR-100 Jan 12, 2019
vgg16-net Add plots for CIFAR-100 Jan 12, 2019
wide-resnet Add plots for CIFAR-100 Jan 12, 2019
.gitignore Add WRNModel Dec 19, 2018
README.md Update README.md Jan 12, 2019
Report.pdf Upload Report Jan 14, 2019
amsgrad.py Use commong padam.py and amsgrad.py Jan 3, 2019
padam.py Use commong padam.py and amsgrad.py Jan 3, 2019
test_error.png Updates Readme Jan 8, 2019
train_loss.png Updates Readme Jan 8, 2019

README.md

Padam: Closing the Generalization gap of adaptive gradient methods in training deep neural networks

Link to paper : Padam: Closing the Generalization gap of adaptive gradient methods in training deep neural networks

You can find our report for the reproducibility challenge here.

Introduction

Adaptive gradient methods such as Adam, Adagrad, Adadelta, RMSProp, Nadam, Adamw, were proposed over SGD with momentum for solving optimization of stochastic objectives in high-dimensions. Amsgrad was recently proposed as an improvement to Adam to fix convergence issues in the latter. These methods provide benefits such as faster convergence and insensitivity towards hyperparameter selection i.e. they are demonstrated to work with little tuning. On the downside, these adaptive methods have shown poor empirical performance and lesser generalization as compared to SGD. The authors try to address this problem by designing a new optimization algorithm that bridges the gap between the space of Adaptive Gradient algorithms and SGD with momentum. With this method a new tunable hyperparameter called partially adaptive parameter \textit{p} is introduced that varies between [0, 0.5].

Setup Dependencies

The recommended version for running the experiments is Python3.

These experiments have been written in tensorflow's eager mode so installing the dependencies is a must to run the code:

  1. Follow the installation guide on Tensorflow Homepage for installing Tensorflow-GPU or Tensorflow-CPU.
  2. Follow instructions outlined on Keras Homepage for installing Keras.

Run a vanilla experiment using the following command at the directory root folder.

python vgg16-net/run.py

Project Structure

The skeletal overview of the project is as follows:

.
├── vgg16-net/
│   ├── run.py  # A script to run the experiments over VGG Net architechture 
│   └── model.py     # VGGNet model
├── resnet18/
│   ├── run.py # A script to run the experiments over ResNet architechture
│   └── model.py     # Resnet 18 model
├── wide-resnet/
│   ├── run.py        #A script to run the experiments over ResNet architechture
│   ├── model.py    # Wide Resnet 18 model
.
folders and files below will be generated after you run the experiment in each model directory
.
├── model_{optimizer}_{dataset}.csv                 # Stores logs for the experiment 
└── model_{optimizer}_{dataset}.h5              # Stores the weights of the final model trained 

Defining Experiment Configuration

You can set the experiment configuration by changing the dictionary in the run.py files. These dictionary contains all the hyperparameter for the each optimizers ie. Adam, Amsgrad, SGD Momentum and Padam.

Experiments

We carry out the experiments to compare the performance of four optimizers - Adam, Amsgrad, SGD Momentum and the proposed algorithm Padam, on 3 modern deep learning architectures ResNet18, WideResNet18 and VGGNet16, over CIFAR-10 and CIFAR-100 datasets. All the experiments have been run for 200 epochs, using categorical cross entropy loss function.

Results

We were sucessful in reproducing the results as predicted in the paper for Cifar-10 and Cifar-100. It is observed that Padam indeed generalizes better than other adaptive gradient method, although it does have a few shortcomings as mentioned in our report. Here, we show the results for VGGNet16, rest of the results have been included in the report.

Results on the CIFAR-10 dataset for VGGNet.

Contributors