Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

T1-T2 Hyperparameter Tuning

Theano implementation of T1-T2 method for tuning continuous hyperparameters. Paper: http://arxiv.org/abs/1511.06727

click me

Currently supporting:

  • architectures: mlp, cnn
  • datasets: mnist, svhn, cifar-10 and not_mnist data sets
  • regularization: batch normalization; various versions of additive and multiplicative gaussian noise; L1, L2, Lmax, soft Lmax penalties; drop-out, per-batch drop-out
  • training regularizers: all gaussian noise, L2, soft Lmax; parametrized per unit, per map (for cnn), per layer, per network
  • optimizers: SGD, momentum, adam
  • T2 gradients: via L-op, via finite difference
  • monitoring: various network activation and parameter statistics, gradient norms and angles

This version was implemented partially as an exercise, more efficient implementation will be developed in keras.

Test performance with initial random hyperparameters vs. same model just tuned hyperparameter:

(different symbols correspond to different experiment setups: varying network architecture, number and degree of freedom of hyperparameters; x-axis: test error before tuning, y-axis: test error after tuning)

About

Theano implementation of T1-T2 gradient-based method for tuning continuous hyperparameters.

Resources

Releases

No releases published

Packages

No packages published

Languages