This is a memory-augmented neural network that uses a neural network as a storage device. Particularly, a controller network provides training examples for a storage network at every timestep. The storage network is trained on these samples with SGD at every timestep (in a differentiable manner). The controller can then query the storage network by feeding it inputs and seeing the corresponding outputs. The end result is that the storage network serves as a memory bank which is "written to" via SGD.
Neural networks seem to provide a lot of desirable properties as memory modules:
- They can store a lot of information.
- They can compress information.
- They can generalize to new information.
- They can interpolate between training samples.
In a sense, a neural network can be seen as a key-value store which tries to generalize to new keys. This seems like the perfect memory structure for a memory-augmented neural network.
Preliminary results on polynomial approximation looked promising. See experiments/poly_approx. After seeing those results, I decided to scale up to a harder meta-learning task.
On the Omniglot handwriting dataset, the model (controlled by a vanilla RNN) outperformed an LSTM in training time (measured in epochs) and accuracy. See experiments/omniglot.