Add leaky ReLUs

[zoq]

Unlike the standard ReL function the leaky rectified linear function has a non-zero gradient over it's entire domain. So instead of having y = max(0, x), you have y = max(x / a, x), where a is some constant. This means you still get some sort of non-linearity, but the gradient can flow through in both directions.

For more information see:

• Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng, "Rectifier Nonlinearities Improve Neural Network Acoustic Models", 2014

Since the parameter is fixed we could add the leakyness factor as a template parameter. The problem with the idea is that C++ doesn't support double as template parameter. So, we need to figure out a way around this issue.

[zoq]

Instead of writing an independent activation function we can just write a new LeakyReLULayer class. The constructor of the LeakyReLULayer class takes the leakyness factor as parameter.

[hercky]

Hi

I'm willing to take this task. I'm new to the community and I'm trying for GSOC 16 and thus I believe implementing this issue can be a good starting point.

The way I see it now, it involves creating a new rectifier function class at ann/activation_functions and then adding that to base_layer.hpp module ( and corresponding test cases at test/activation_functions_test.cpp )

[zoq]

You are right this is a good starting point to get familiar with the code.

The BaseLayer only works with activation functions that can be called without any additional parameters like the sigmoid or the tanh function. Since the leaky rectified linear function uses the leakyness factor as an additional parameter you can't use the BaseLayer to call the function. But there is an easy solution you can directly implement the LeakyReLULayer without implementing the activation function in ann/activation_functions first. The LeakyReLULayer should have the same functions as SoftmaxLayer but should allow the specification of the leakyness factor in the constructor.

Please leave a comment if something doesn't make sense.

[abhinavchanda]

@zoq , I have written a LeakyReLULayer class here . I have used the forward and backpropagation functions similar to those in the base_layer.hpp. Please let me know if any changes are required. Also, can you guide me as to how to write test cases for this layer.

[zoq]

Thanks for the contribution. Before I merge the code in (I guess you will open a pull request) could you take a look at the design guidelines especially the comments section:

https://github.com/mlpack/mlpack/wiki/DesignGuidelines

It's minor, but I tend to be picky about code but I am not mean. :)

It would also be great if you could combine the two constructors into one:

LeakyReLULayer(const double alpha = 0.01) : alpha(alpha)

And last but not least, can you add a function that returns alpha and enables the modification of alpha?

[zoq]

About the test, take a look at the activation_functions_test.cpp, it basically tests different activation functions with edge cases and compares them with manually calculated data.

[abhinavchanda]

Hi. Thanks for the suggestions. I have made the required changes here.Regarding testing I have a doubt. As leakyReLu is a layer, as opposed to a single neuron, so it should have only forwad and backward as public methods and the activation function and derivative should not be exposed, but in tests/activation_functions_test.cpp only activation functions and their derivatives are being tested.

[GYengera]

@abhinavchanda your code is well written. It helped me to understand the codebase better.
I think you need to serialize the layer at line 156.
template <typename Archive> void Serialize(Archive& ar, const unsigned int) { }

[sharathts]

@zoq Is this task still open?

[zoq]

@sharathts No, the code was merged in e6f7ffe.

[sharathts]

@zoq Thank you for the information.