This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
Fix RMSProp update rule #6235
Merged
Fix RMSProp update rule #6235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Follow the formula presents in Alex's paper, this prevents taking square root of a negative value (caused by arithmetic error).
|
You need to fix the tests in tests/python/unittests/test_optimizer too |
|
Although I do not know how to write Perl, I also updated the test in perl module. (Hope it will pass) |
|
@sifmelcara thank you for updating perl test. |
saurabh3949
pushed a commit
to saurabh3949/mxnet
that referenced
this pull request
May 23, 2017
* Fix RMSProp update rule Follow the formula presents in Alex's paper, this prevents taking square root of a negative value (caused by arithmetic error). * Fix the formula of non centered version of RMSProp * Fix RMSProp update rule in python test * Fix RMSProp update rule in perl test
Guneet-Dhillon
pushed a commit
to Guneet-Dhillon/mxnet
that referenced
this pull request
Sep 13, 2017
* Fix RMSProp update rule Follow the formula presents in Alex's paper, this prevents taking square root of a negative value (caused by arithmetic error). * Fix the formula of non centered version of RMSProp * Fix RMSProp update rule in python test * Fix RMSProp update rule in perl test
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using Alex's version of RMSProp to train a network, I found it sometimes update some network weights to NaN.
After that, I found the RMSProp implementation is a little bit different from Alex's paper. In equation 40, the epsilon should be under the square root.
I guess I got
NaNbecausestate_n - state_g * state_gcan sometimes be something like-1e-20(because of floating point arithmetic error). Indeed, after I applied this patch, it stops update network weights toNaN.