Any plan to add SoftClipping and SmoothMax

[dmilos]

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow version (you are using):
• Are you willing to contribute it (Yes/No):
  Yes.
  Describe the feature and the current behavior/state.
  SoftClipping for activation. [https://en.wikipedia.org/wiki/Activation_function]
  SmootMax for pooling. [https://en.wikipedia.org/wiki/Smooth_maximum]
  Will this change the current api? How?
  No. This is new feature.
  Who will benefit with this feature?
  Everybody.
  SmoothClipping is smoot function and can not suffer from vanishing/exploding gradients.
  So with this we will have faster learning. Create Soft Clipping Layer tiny-dnn/tiny-dnn#1014. Only 10 layers achieve 99.35% on MNIST set, Comparing to original 12 and 99.0%
  Any Other info.

[rmothukuru]

@dmilos,
Sorry for the delayed response. Relu Activation Function seems to be good for handling the problem of Vanishing and Exploding Gradients.

Can you please let us know how efficient are the activation functions that you proposed compared to the existing non-saturated activation functions like Relu, Elu, Selu? Thanks!

[dmilos]

SmoothClipping is smooth function and can not suffer from vanishing/exploding gradients.
So with this we will have faster learning. tiny-dnn/tiny-dnn#1014. Only 10 layers achieve 99.35% on MNIST set, Comparing to original 12 and 99.0%

[dmilos]

For more details in depth for Soft Clipping see:
Neural Network-Based Approach to Phase Space Integration
M. D. Klimek1,2*, M. Perelstein1
1 Laboratory for Elementary Particle Physics, Cornell University, Ithaca, NY, USA
2 Department of Physics, Korea University, Seoul, Republic of Korea

• klimek@cornell.edu
  August 18, 2020

[github-actions]

This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you.

[dmilos]

Appears that answer is: no.
Thanks.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you.