General purpose Hessian-free optimization in Theano
Switch branches/tags
Nothing to show
Latest commit 8496f8d Jun 26, 2014 @boulanni added license
Failed to load latest commit information.
LICENSE.txt added license Jun 26, 2014
README.txt added license Jun 26, 2014 structural damping fix suggested by Sigurd Spieckermann May 13, 2014


I wrapped my Hessian-free code in a generic class, usable as a black-box to train your models if you can provide the cost function as a Theano expression.

It includes all the details in Martens (ICML 2010) and Martens & Sutskever (ICML 2011) crucial to make it work:
- Tikhonov damping with the Levenberg-Marquardt heuristics,
- Gauss-Newton matrix products (you specify an Theano expression `s` to section your computational graph in 2),
- Proper handling of batches and mini-batches (an example SequenceDataset class is provided for variable-length input)
- Conjugate gradient (CG) with information sharing, backtracking, preconditioning and terminations conditions.
- Structural damping for RNNs.

It relies heavily on the Rop. In practice, I could make it work without hassle for a feed-forward network, an RNN with different objectives, NADE (Larochelle) and a more complex model (RNN-NADE) that ties two scans together, so it seems reasonably flexible.
Only the gradients and Gauss-Newton matrix products (95% of the computation) are in Theano, CG and the training logic is in python. It runs on GPU, but for the models I tried, it was a bit slower.
Hessian-free is slow, you need CG batch sizes of 1000+ (don't skimp on this), but you can get really better results than SGD from it with almost zero tweaking.

There is an option to save and recover a checkpoint of training and do early stopping.

I included an RNN example that can memorize an input for 100 time steps (example_RNN). Launch it on 4 cores, come back in 8 hours, and you should have at least one nice solution with 0 error on the validation set.
In comparison, SGD can solve this problem about 0.0% of the time.

It is available here:

If you use this software for academic research, please cite the following paper:

[1] N. Boulanger-Lewandowski, Y. Bengio and P. Vincent, "Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription", Proc. ICML 29, 2012.

Author: Nicolas Boulanger-Lewandowski
University of Montreal, 2012