Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReLU node causes numeric instability #14

Closed
emjotde opened this issue Sep 20, 2016 · 1 comment
Closed

ReLU node causes numeric instability #14

emjotde opened this issue Sep 20, 2016 · 1 comment
Assignees
@emjotde
Labels
bug

Comments

@emjotde
Copy link
Member

emjotde commented Sep 20, 2016

When running the MNIST example with ReLU for more than 25 epochs NAN appears. This seems to be a known issue with ReLU and crossentropy, and the suggested solution is some sort of clipping. This requires more investigation.
@emjotde emjotde added the bug label Sep 20, 2016
@emjotde emjotde self-assigned this Sep 20, 2016
@emjotde
Copy link
Member Author

emjotde commented Sep 21, 2016

Replacing softmax with log-softmax in the cross-entropy layer fixed this.

@emjotde emjotde closed this as completed Sep 21, 2016
@520jefferson 520jefferson mentioned this issue Aug 22, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@emjotde emjotde

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@emjotde