Implementation of different first order (FO) stochastic methods: stochastic gradient descent (SGD), stochastic gradient descent with momentum (SGDw/Momentum) , Root Mean Squared Propagation (RMSProp), Adaptive Gradient Algorithm (Adagrad). This project was made to test these different optimizers on a handwritten digit classifier using the well-known MNIST dataset. This dataset has 60000 training images, and 10000 test images. Each image is of size 28 × 28 pixels, and shows a digit from 0 to 9.
For a more detailed explanation of the algorithms mentioned above, please read Exercise instructions.pdf, it contains also some theoretical questions answered in Answers.pdf (handwritten).
The project was part of an assignment for the EPFL course EE-556 Mathematics of data: from theory to computation. The backbone of the code structure to run the experiments was already given by the professor and his assistants, what I had to do was to implement the core theoretical concepts to actually make the experiments work. Hence, every code file is a combination of my personal code and the code that was given us by the professor.
The following table shows a summary of the training done by running main.py in the code folder.
The following image shows an example of the output figures of the code main.py. All plots show a comparison of convergence speed to the optimal loss and accuracy, for different learning rate's values.
Download this repository as a zip file and extract it into a folder The easiest way to run the code is to install Anaconda 3 distribution (available for Windows, macOS and Linux). To do so, follow the guidelines from the official website (select python of version 3): https://www.anaconda.com/download/
Additional package required are:
- matplotlib
- pytorch
To install them write the following command on Anaconda Prompt (anaconda3):
cd *THE_FOLDER_PATH_WHERE_YOU_DOWNLOADED_AND_EXTRACTED_THIS_REPOSITORY*Then write for each of the mentioned packages:
conda install *PACKAGE_NAME*Some packages might require more complex installation procedures (especially pytorch). If the above command doesn't work for a package, just google "How to install PACKAGE_NAME on YOUR_MACHINE'S_OS" and follow those guides.
Finally, run main.py.
python main.py-
code/: folder containing all the code to run the training and testing, included the optimization algorithms
-
code/main.py: main script to be run in order to do the training and output the plots.
-
Answers.pdf: pdf with the answers and plots to the assignment of the course
-
Exercise instructions.pdf: pdf with the questions of the assignment of the course
Python, Pytorch, Matplotlib. Machine learning, artificial neural networks, multilayer perceptron (MLP), implementation of optimization methods.
