Augmented RNN memory via live SGD
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.


This is a memory-augmented neural network that uses a neural network as a storage device. Particularly, a controller network provides training examples for a storage network at every timestep. The storage network is trained on these samples with SGD at every timestep (in a differentiable manner). The controller can then query the storage network by feeding it inputs and seeing the corresponding outputs. The end result is that the storage network serves as a memory bank which is "written to" via SGD.


Neural networks seem to provide a lot of desirable properties as memory modules:

  • They can store a lot of information.
  • They can compress information.
  • They can generalize to new information.
  • They can interpolate between training samples.

In a sense, a neural network can be seen as a key-value store which tries to generalize to new keys. This seems like the perfect memory structure for a memory-augmented neural network.


Preliminary results on polynomial approximation looked promising. See experiments/poly_approx. After seeing those results, I decided to scale up to a harder meta-learning task.

On the Omniglot handwriting dataset, the model (controlled by a vanilla RNN) outperformed an LSTM in training time (measured in epochs) and accuracy. See experiments/omniglot.