Various Optimizers based on Gradient Descent

• Final update: 2018. 12. 15.
• All right reserved @ Il Gu Yi 2018

Educational Purpose

• Implementation various optimization algorithms based on gradient descent
• Only use numpy, don't use deep learning framework like TensorFlow
• Low level coding in each algorithm

Getting Started

Prerequisites

• Python 3.6
  • numpy, matplotlib
• Jupyter notebook
• OS X and Linux (Not validated on Windows but probably it might work)

Contents

Linear Regression using Gradient Descent

• Gradient Descent
  • 01.gradient.descent.ipynb
• Stochastic Gradient Descent
  • 02.stochastic.gradient.descent.ipynb
• Coordinate Gradient Descent
  • 03.coordinate.gradient.descent.ipynb
• Linear Regression with 5 optimizers
  • regression.all.opt.ipynb

Optimization of Beale Function using Various Gradient Descent Algorithms

• Gradient Descent
  • 04.1.sgd.ipynb
  • 04.2.sgd.ipynb
• Momentum
  • 05.momentum.ipynb
• Nesterov Momentum
  • 06.1.Nesterov.momentum.ipynb
  • 06.2.Nesterov.momentum.ipynb
• Adagrad (paper link)
  • 07.adagrad.ipynb
• RMSprop (Tieleman and Hinton's Lecture slide, 2012)
  • 08.RMSprop.ipynb
• Adam optimizer (arXiv:1412.6980)
  • 09.adam.ipynb
• optimizer implemented by TensorFlow
  • optimizer.tf.version.ipynb
• optimizer implemented by TensorFlow for plot
  • optimizer.tf.all.opt.plot.ipynb

Results

Optimization of Linear Regression using Various Gradient Descent Algorithms

Optimization of Beale Function using Various Gradient Descent Algorithms

References

• S. Ruder, An overview of gradient descent optimization algorithms (arXiv:1609.04747)
• S. Ruder's blog: An overview of gradient descent optimization algorithms (the same content as above)
• Louis Tiao's blog: Visualizing and Animating Optimization Algorithms with Matplotlib
  • Some codes for 3D plot and contour plot are borrowed from this blog

Author

Il Gu Yi