Add hard tanh layer

[zoq]

The hard tanh function is sometimes preferred over the tanh function since it is computationally cheaper. It does however saturate for magnitudes of x greater than 1. The activation of the hard tanh is:

f(x) = max, if x > max,
f(x) = min, if x < min,
f(x) = x, otherwise.

It would be great, if mlpack would provide an implementation of a HardTanHLayer that allows the specification of the min and max value.

HardTanHLayer(const double minValue = -1, const double maxValue = 1)

[arasuarun]

I'd like to take this up. Along with the HardTanhLayer, there needs to be a HardTanhFunction, right? Thanks.

[zoq]

You can directly implement the HardTanhLayer. The extra activation functions (e.g. the tanh function) are supposed to be used in combination with the BaseLayer. The BaseLayer only works with activation functions that can be called without any additional parameters like the sigmoid or the tanh function. Since the HardTanhLayer should allow the specification of the min and max value as additional parameter you can't use the BaseLayer. The HardTanhLayer should have the same functions as SoftmaxLayer.

[arasuarun]

Okay, got it. I'll check it out and start right away. Thanks.

[vtantia]

Hello, I was trying to understand the code to add this function. I was also reading bias_layer.hpp. On line 156,157 I cannot understand(after extensive googling) what data::createNVP is doing, and what an NVP is? Could you help me with that? Will enable me to understand better

[rcurtin]

Hi there Vinayak,

The data::CreateNVP() function is used in place of BOOST_SERIALIZATION_NVP(); see here:
https://github.com/mlpack/mlpack/blob/master/src/mlpack/core/data/serialization_shim.hpp#L64
There's a little more documentation in that file, and you could also see this section of the design guidelines:
https://github.com/mlpack/mlpack/wiki/DesignGuidelines#requirements-of-each-class

I hope this is helpful. Let me know if I can clarify anything.