No description or website provided.
Jupyter Notebook Python
Latest commit 9656c3a Jul 15, 2016 @charlesmartin14 charlesmartin14 committed on GitHub Delete why-dl-works.pdf


This is an attempt to use TensorFlow to reproduce the Big, Deep, Simple MLP

The idea is simple. They claim to get state-of-the-art accuarcy on MNIST using a 5-layer MLP, but running a large number of epochs with just SGD, a decaying learnng rate, and an augmented data set.

This paper still appears to be 1 of the top 10 results in deep learning

The key idea is that the augmented data set can provide, in practice, an infinite amount of training data.

I am trying to repeat this experiment using Tensorflow and 2 kinds of augmented data sets.


We will use 2 different data augementation methods


Install infimnist and set the path in


Download alignmnist.npz

Test with

  • AlignMNIST-test.ipynb

Run these notebooks next

run this notebook first to create and test

  • AugmentMNIST.ipynb


  • 2-Layer-MLP-AlignMNiST.ipynb

  • 2-Layer-MLP-InfiMNiST.ipynb

Results so far:

I have run 75 epochs of AlignMNIST using RMSProp for a 2 layer and 5 layer MLP

the resulting accuracies are much lower than reported in the paper

The initial result suggest that having a very good augmentation scheme matters, as the AlignMNIST data is much more generalizable than the InfiMNIST data. Although one has to be careful to look for information leakage, and it would be good to double check that the AlignMNIST does not actually include deformations of the test data.