Tests for Stochastic Optimization

[zoq]

mlpack implements a some gradient-based optimization algorithms; each method is tested on some simple function to test they work as expected for that particular function.

Since implementing meaningful tests is probably the most underrated task which often takes more time as implementing the method itself, it's 'fine' if the optimization algorithms are able to solve some simple functions. The goal of this ticket is to implement a conglomerate of some additional functions to test the optimization algorithms.

In "Unit Tests for Stochastic Optimization" Tom Schaul et al. proposed a bunch of really interesting basic functions and combinations that we could use as a basis for this ticket. A start could be the implementation of some simple generator e.g. a simple Line function generator that can be combined to create other functions:

// Line function generator.
class LineFunction
{
 public:
  //! Nothing to do for the constructor.
  LineFunction() { }

  /*
   * Construct the line function generator with the given parameters.
   *
   * @param xs Start point of the function.
   * @param xe End point of the function.
   * @param fs Start point value.
   * @param fd Start point derivative.
   */
  LineFunction(const arma::mat& xs,
               const arma::mat& xe,
               const arma::mat& fs,
               const arma::mat& fd) { }
  //! Evaluate a function.
  double Evaluate(const arma::mat& x) const;

  //! Evaluate the gradient of a function.
  void Gradient(const arma::mat& x, arma::mat& gradient) const;
};

#include "line_function.hpp"

using namespace mlpack;
using namespace mlpack::optimization;
using namespace mlpack::optimization::test;

double LineFunction::Evaluate(const arma::mat& x)
    const
{
  arma::vec temp = (x - xs) * fd + fs;
}

...

[kris-singh]

I would like to work on this one. I have started reading the paper. I would create a PR soon.

[kris-singh]

Read the paper kind of understood it. So, I make any error surface using now the combination fo these basic shapes right. And when I run let's say a sgd on it. Then which evaluate function should called.(Does it depend on xs <= x <= xe). After that we evaluate the loss on this local function and the take the step in the gradient of this local function is that it.

[zoq]

You are right, right now we have some simple functions to test the optimization algorithms like:

https://github.com/mlpack/mlpack/blob/master/src/mlpack/core/optimizers/sgd/test_function.hpp
https://github.com/mlpack/mlpack/blob/master/src/mlpack/core/optimizers/aug_lagrangian/aug_lagrangian_function.hpp
https://github.com/mlpack/mlpack/blob/master/src/mlpack/core/optimizers/gradient_descent/test_function.hpp

which are great, but this ticket would allow us to define a bunch of different test function just by combining different function generators. Let's say you like to test it on a non-convex gaussian line function; All you need is a Line function generator and another generator that creates the non-convex part of the gaussian function.

[sinjan25]

@zoq Can I work on this one? No one has made a PR yet.

[kris-singh]

@sinjan25 you can work on this too. I will be implementing the LineFunction Shape Prototypes. Maybe you could do some NonConvex Shape Prototype.

[sinjan25]

@zoq @kris-singh Alright! I will then be working on a genarator that creates the non-convex part of the gaussian function.

[kris-singh]

@zoq i implemented the line function. I wanted to now we would write a decomposable function. Which would have activate the function based upon the x1 given right. so we would need to check every time. can this checking be done in compile time. Or how should i go about implementing the decomposable function. Also in sgd implementation we have been given the iterate(which basically can be training points or functions) but what I dont understand is if these are function, then what are the points these functions are evaluated on.

[kris-singh]

@zoq can you answer this. Thanks

[zoq]

We have to provide a Function that is able to combine different generators and that provides the same interface as e.g. the RosenbrockFunction function. x1 is a single input point or vector that has to be specified at build time. Right now we use GetInitialPoint() that comes with each function and provides some input for the evaluation.

I'm not sure what you mean with checking of x1, can you elaborate further?

[shikharbhardwaj]

Hi!
Is there anything I could implement to help with the progress of this issue?

[zoq]

Hello @shikharbhardwaj there are a couple of different and interesting generators that could be implemented, so feel free to work on this issue.

[shikharbhardwaj]

Thanks for the reply @zoq.

How exactly would the generator interface look?

The line function interface you posted generalises the line function over the arbitrary number of dimensions the user wishes to use. The way I was thinking about the problem was to define the function prototypes in a single dimension and then use something like a FunctionFactory to combine those single dimension functions as described in the paper and implemented in the reference implementation here.

Any thoughts?

[zoq]

Covered by: #1151