Find file History
Pull request Compare This branch is 124 commits behind exacity:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

SGD Comparison

Test various SGD algorithms on logistic regression and MLP, including

  • vanilla SGD
  • Momentum
  • Nesterov Accelerated Gradient
  • AdaGrad
  • RMSProp
  • AdaDelta
  • Adam
  • Adamax

The relation of these algorithms is shown in the following figure (my personal view). relation

This code is based on Theano, please install relative packages. The implementation of logistic regression and MLP is based on the Theano tutorial.

Test results

We measure the performance of these SGD algorithms by comparing the training curve and validation error.

Logistic Regression




For more details about these algorithms, please refer to my blog (Chinese).