Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Gradient-based Optimization of Hyperparameters through Reversible Learning

Source code for


Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

Authors: Dougal Maclaurin, David Duvenaud, and Ryan P. Adams

How to run these experiments

You'll need to install autograd, our automatic differentiation package. However, autograd (aka funkyYak) has changed a lot since we wrote the hypergrad code, and it would take a little bit of work to make them compatible again.

However, the hypergrad code should work with the version of FunkyYak as of Feb 2, at this revision:

So if you clone autograd, then type git checkout be470d5b8d6c84bfa74074b238d43755f6f2c55c, you should be at the same version we used to run the experiments.

That version also predates the file, so to get your code to use the old version, you'll either have to copy into the old revision and reinstall, or add FunkyYak to your PYTHONPATH.

Feel free to email us with any questions at (, (

For a look at some directions that didn't pan out, take a look at our early research log.


Exploring differentiation with respect to hyperparameters






No packages published