More advanced activation functions.

[WuZhuoran]

• ReLU
• Tanh
• Affine
• Sigmoid
• Leaky ReLU
• ELU
• Linear
• Softmax
• Hard Sigmoid
• Exponential
• PReLU
• ThresholdedReLU
• SELU
• Softplus
• Softsign

I will try to implement rest part of activation functions. Any better ideas for activation function?

[kmario23]

@WuZhuoran Great! I think it's also worth looking into Activation_function#Comparison_of_activation_functions

[ddbourgin]

Awesome! Two quick notes:

• Linear is just a special case of Affine (with slope=1, intercept=0), so no need to implement a separate function
• Softmax actually already exists, but as a Layer rather than an activation function. I decided to do this because it is much more efficient to calculate the gradient through the activation and projection weights together rather than in sequence.

Otherwise, looks good! I'll have a look at your PR when I have a spare moment :)

[WuZhuoran]

@ddbourgin Cool.
Fixed.

I will commit after you review the code. About the NumPy Style Python Docstrings, I will update ASAP.