Find file History
Pull request Compare This branch is 124 commits behind exacity:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
figures
LogisticRegression.py
MLP.py
README.md
algorithms.py
sgd_comparison.py

README.md

SGD Comparison

Test various SGD algorithms on logistic regression and MLP, including

  • vanilla SGD
  • Momentum
  • Nesterov Accelerated Gradient
  • AdaGrad
  • RMSProp
  • AdaDelta
  • Adam
  • Adamax

The relation of these algorithms is shown in the following figure (my personal view). relation

This code is based on Theano, please install relative packages. The implementation of logistic regression and MLP is based on the Theano tutorial.

Test results

We measure the performance of these SGD algorithms by comparing the training curve and validation error.

Logistic Regression

LR

MLP

MLP

For more details about these algorithms, please refer to my blog (Chinese).