Different activation functions

[philiiiiiipp]

Hey,

first of, cool work :-)! As far as I see it brain.js is currently using a sigmoid/logistic activation function. I think it would be quite beneficial to.

1. Document that this one is used
2. Create different ones

I could implement some of them like tanH which are working in the range of (0,1) which is currently used as output.

[robertleeplummerjr]

I'm currently working on implementing tanH in the recurrent branch:
Forward propigate a matrix: https://github.com/harthur-org/brain.js/blob/recurrent/lib/recurrent/matrix/tanh.js
Backpropagate a matrix: https://github.com/harthur-org/brain.js/blob/recurrent/lib/recurrent/matrix/tanh-b.js

Currently is 90% or so working (weights aren't zeroing out and result in infinity after a few runs) and will be done soon (maybe a week or two?) I think after I make headway there, I can address this.

[robertleeplummerjr]

Just fyi, have not forgotten. A minor surgery and a bunch of planning later, this is still on the board.

[robertleeplummerjr]

Also, pull requests are always welcome.

[robertleeplummerjr]

Booya! https://github.com/harthur-org/brain.js/tree/other-activation-functions
Doesn't yet work, but the most of the methods and math is there.

[robertleeplummerjr]

#87

Would you be so kind as to give a code review and any insights?

[robertleeplummerjr]

example:

const net = new brain.NeuralNetwork({
  activation: 'leaky-relu'
});