Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NaN detected #64

Open
amirj opened this issue Oct 13, 2016 · 1 comment
Open

NaN detected #64

amirj opened this issue Oct 13, 2016 · 1 comment

Comments

@amirj
Copy link

amirj commented Oct 13, 2016

After some iterations the following error is occurred:

...
128 samples computed
256 samples computed
384 samples computed
462 samples computed
Valid 2.78936
Epoch 0 Update 88100 Cost 0.587470054626 UD 1.63904905319
NaN detected

Reloading the model using reload=True parameter, continue the learning process.
It maybe related to this issue.
@amirj amirj changed the title Nan is detected Nan detected Oct 13, 2016
@amirj amirj changed the title Nan detected NaN detected Oct 13, 2016
@valentinkoe
Copy link

Have you tried @orhanf 's suggestions in the issue you linked? For my use case it seems like applying gradient clipping (setting clip-c to 1.0) solved the problem of NaNs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amirj @valentinkoe