This is an attempt to use TensorFlow to reproduce the Big, Deep, Simple MLP
The idea is simple. They claim to get state-of-the-art accuarcy on MNIST using a 5-layer MLP, but running a large number of epochs with just SGD, a decaying learnng rate, and an augmented data set.
This paper still appears to be 1 of the top 10 results in deep learning
The key idea is that the augmented data set can provide, in practice, an infinite amount of training data.
I am trying to repeat this experiment using Tensorflow and 2 kinds of augmented data sets.
We will use 2 different data augementation methods
Install infimnist and set the path in infimnist.py
Run these notebooks next
run this notebook first to create and test augmentmnist.py
Results so far:
I have run 75 epochs of AlignMNIST using RMSProp for a 2 layer and 5 layer MLP
the resulting accuracies are much lower than reported in the paper
The initial result suggest that having a very good augmentation scheme matters, as the AlignMNIST data is much more generalizable than the InfiMNIST data. Although one has to be careful to look for information leakage, and it would be good to double check that the AlignMNIST does not actually include deformations of the test data.