This repo contains implementations of various optimization algorithms for machine-learning applications which were implemented as part of a course about optimization.
The optim/optimizers.py module provides implementations of stochastic gradient descent (SGD),
Nesterov accelerated gradient (AGM), and stochastic variance-reduced gradient
(SVRG).
The optim/projections.py module provides algorithms for projection of points
onto the (scaled) Simplex (using either a regular euclidean norm or a custom
norm induced by a positive definite matrix) and also an algorithm for projecting
matrices onto the set of matrices with a bounded nuclear-norm.
The linreg package contains functions to generate linear-regression datasets,
run experiments using the above optimization methods and plot results.
The matcomp/data.py module contains functions to create datasets for the
matrix-completion tasks, including synthetic data and data from the MovieLens
100k/1M datasets.
The matcomp/models.py module implements a few algorithms for solving the
matrix completion task. For example, optimizing over a projection onto the space of low-rank
matrices, factorizing into a product of two low-rank matrices and optimizing
over them, and optimizing over over a projection onto a bounded nuclear-norm.
The orps/data.py module contains functions to create a small dataset for the ORPS
task.
The orps/models.py module contains implementation of multiple algorithms to
solve the ORPS problem, as well as calculation of the regret and wealth per trading round.
The implemented algorithms are online gradient descent (OGD), regularized
follow-the-leader (RFTL), Newton-step, best-fixed portfolio and best
single-asset portfolio.