For the Computational Mathematics for Learning and Data Analysis course we expanded the functionality of Neuradillo. Using backpropagation algorithm to compute the gradient, we developed L-BFGS (following this paper) and Proximal Bundle optimization algorithms to train feedforward neural network.
More information about the theory and the convergence analysis regarding the algorithms can be found in the report.
The project use CMake 3.16 as building system, Conan as package manager and GCC as compiler. The project has two types of builds: Debug and Release. The former can be used to debug the program if more functionalities need to be added, the latter is used to train the network and to obtain the benchmark values with all the code optimization given by the compiler and the OpenMP library.
- Clone the following repo: https://github.com/darcamo/conan-armadillo
- Inside the cloned repo run:
conan create . armadillo/stable - If Armadillo is installed correctly an example program is executed and you can start using it through Conan.
- Check that the Armadillo version of the library installed through Conan is the same you have inside the
conanfile.txtof the project. In our case the version is 9.800.2 but the Armadillo version downloaded through Conan can be different due to the remote repository update.
The project uses Gurobi as an off-the-shelf solver for the Master Problem of the bundle method, it can be downloaded from here. This library is automatically linked to the project by CMake, but to do that few operations need to be made. Basically, CMake find Gurobi by looking at the file FindGUROBI.cmake, inside it you need to set the variable GUROBI_HOME to the path that leads to the folder where you have download Gurobi with the license of it in the upper folder. The license can be downloaded for free from the Gurobi website if you are a university student/professor.
For example, in our case the path to Gurobi is set as: /opt/gurobi902/linux64 and the license is saved at /opt/gurobi902.
To use different optimizers you need to pass inside the Network public method SetOptimizer(Optimizer*) an object that has the type of the optimizer that you want to use. In the main.cpp file there is already written the three objects that you need to pass to the Network in order to use a specific optimizer. You need to uncomment one and be sure that the method is called before the Train() method starts. To use the Nesterov momentum you need to call the SetNesterov() method and pass to it a true boolean value. For the Momentum and the Nesterov Momentum, the Network object accepts in the method Train() a parameter that specifies their value at each iteration of the algorithm.
Instead, the regularizations terms are set by calling the method SetRegularizer() by specifying the L1 or L2 as input and their values is also specified as parameter passed in the Train() method.
The constructor of the L-BFGS object accepts as first parameter the number of past curvatures to store and as second parameter a seed value makes the line search procedure repeatable. Also, the constructor of the Proximal Bundle Method accepts as parameter the percentage of the past subgradient that are not considered by the algorithm at each iteration.
If CMake and Armadillo were installed correctly you have to create a directory where CMake store the configuration files needed to run the project (Gurobi installation is not required unless you want to run Proximal Bundle method). So first we create the folder to store the file generated by CMake:
mkdir build && cd build
Inside the build folder to generate the files that are needed by Conan type:
conan install ..
Then the for the CMake files give:
cmake .. -DCMAKE_BUILD_TYPE=Release/Debug
At the end to build the project type:
cmake --build .
If all the process is done correctly a /bin folder is created with a binary file inside. This file can be executed with:
./main
The Error (plus the regularization term), Gradient Norm and Computational time values on the training set are printed during the execution.
To obtain the plots used in the report open Gnuplot in the folder docs/report/data/Comparison and load the scripts with
load "<script-name.gp>"
Since the comparisons were made on the MONK's datasets the Gnuplot script files are all saved inside the Comparison folder inside the Monk1, Monk2 and Monk3 folders. All the data used to obtain the plots are saved inside the folder docs/report/data. Each algorithm has a folder that contains all the data obtained in Monk1, Monk2 and Monk3 on M1 and M2. For example the data obtained and the Gnuplot scripts for L-BFGS on Monk1 are stored inside the folder docs/report/data/LBFGS/Monk1 .
Here we show some plots we obtained during the optimization phase.
The project has the following structure:
More optimizers can be added by creating a class that derives from the Optimizer abstract class that implement the pure virtual methods OptimizeBackward(Network*) and OptimizeUpdateWeight(Network*). After that is sufficient to create the object Network and pass the new optimizer by using the method SetOptimizer(Optimizer*) before calling the public Network method Train() with all the parameters.
This project was developed for the course of Computational Mathematics for Learning and Data Analysis at the University of Pisa under the guide of Prof. Antonio Frangioni and Prof. Federico Poloni.
- Giovanni Sorice - Giovanni Sorice
- Francesco Corti - FraCorti