Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tanh layer for badly scaled input values #1455

Merged
merged 2 commits into from
Nov 26, 2014
Merged

Fix tanh layer for badly scaled input values #1455

merged 2 commits into from
Nov 26, 2014

Conversation

seanbell
Copy link

The current tanh layer uses an expression that is unstable for large values. It returns NaN if the input is badly scaled (outside the range [-40, 40] for single precision float).

This fix uses the tanh library function, which is much more stable. I also added a unit test (there was previously no test for tanh) that checks the case of large input values. It compares against an exact expression for tanh that also handles overflow. The unit test fails if you use the previous naive implementation.

@seanbell seanbell changed the title fixed tanh for input values outside the range [-40, 40] Fixed tanh layer for badly scaled input values Nov 19, 2014
@seanbell seanbell changed the title Fixed tanh layer for badly scaled input values Fix tanh layer for badly scaled input values Nov 19, 2014
@jeffdonahue
Copy link
Contributor

Great, thanks for the fix @seanbell! I've run into this problem and had hackily worked around it by just calculating it as tanh(x) = 2 * sigmoid(2*x) - 1...given that it's apparently a library function, it makes sense to use that though. I'll merge this.

jeffdonahue added a commit that referenced this pull request Nov 26, 2014
Fix tanh layer for badly scaled input values
@jeffdonahue jeffdonahue merged commit a35929f into BVLC:dev Nov 26, 2014
@seanbell seanbell deleted the tanh-fix branch November 29, 2014 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants