Bipolar sigmoid added #3506
Bipolar sigmoid added #3506
Conversation
|
Hey @zoq I had checked out the issue you had mentioned, I just wanted to ask about how we can tackle functions which do require both the x and f(x) value for the calculation of their derivatives ,a great example for this type of activation function is the EliSH function whose derivative can't be expressed solely in terms of f(x).(I have even checked other libraries and tried my self to solve for f'(x) in terms of f(x) but was not able to find any implementation that uses only f(x) values.) A possible fix for these kinds of layers is to store the x values as a member of the class and use them in the deriv() method pls do let me know if there is any problem with this implementation or if there are any other fixes possible I will look into it. |
|
Yes, you have ti implement it as a layer and store the activation function in the layer. |
|
@zoq Ok I will get to implementing all the required ones as layers. Then in this PR, I have fixed all activation functions which do have an inverse or can be brought into terms of their derivatives and add the jacobian test for all of them and in the next few PRs I will add the other ones as a layer . Please do let me know if there are any problem in this. |
|
The jacobian testing implemented is as same as given in #3478 |
|
Hey @zoq can you please review the changes and check if the above mentioned plan is fine so that I can get to implementing the non invertible functions as layers |
Looking at it right now. |
src/mlpack/methods/ann/activation_functions/bipolar_sigmoid_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/bipolar_sigmoid_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/bipolar_sigmoid_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/bipolar_sigmoid_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/bipolar_sigmoid_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/hyper_sinh_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/hyper_sinh_function.hpp
Outdated
Show resolved
Hide resolved
src/mlpack/methods/ann/activation_functions/hyper_sinh_function.hpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Can you update https://github.com/mlpack/mlpack/blob/master/HISTORY.md as well?
![mlpack-bot[bot]](https://avatars.githubusercontent.com/in/23934?s=60&v=4){kind=small}
|
Hello there! Thanks for your contribution. I see that this is your first contribution to mlpack. If you'd like to add your name to the list of contributors in In addition, if you'd like some stickers to put on your laptop, I'd be happy to help get them in the mail for you. Just send an email with your physical mailing address to stickers@mlpack.org, and then one of the mlpack maintainers will put some stickers in an envelope for you. It may take a few weeks to get them, depending on your location. 👍 |
|
Let's rerun the CI. |
|
Hey @zoq can this be merged so that I can create the subsequent pull requests required |
|
Sorry for the delay, thanks again for the contribution. |
Implemented the bipolar sigmoid activation function
Fixed invertible activation functions
Added jacobian testing for all of the above mentioned ones
Tested and used in various research papers:
https://www.scitepress.org/Papers/2018/100372/100372.pdf
https://iopscience.iop.org/article/10.1088/1742-6596/1255/1/012023/meta