-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.1.0 changes for ranger_adabelief #19
Comments
Hi @bratao, thanks for asking. Currently don't have a plan for a big change with the ranger version, will fix some small errors. I don't have much experience with ranger version, also the decoupled weight decay and rectification is turned on by default in ranger (these two defaults are modified in adabelief-pytorch), except the eps is set as 1e-5 in ranger. Do you have a feeling what is a good default value for eps in ranger-adabelief? Or any other ideas on potential improvements? |
Hi, I would recommend setting the default value of 'weight decoupling' to true. If you know any counterexamples where coupling helps, please, show a link, |
@dvolgyes Thanks for feedback, actually for all latest implementations the default for weight_decouple is True. BTW, since you mentioned decoupled decay is enabled in Adam in PyTorch, I checked the source code it seems Adam does not have much change, am I missing something? Could you specify where in Adam of PyTorch is decoupled weight decay enabled? Thanks a lot |
Hi, It depend how you see it. Check out the documentation of Adam between version 1.5.1 and 1.6: The old one was: The new one: I did not go into the details checking how they implemented it, but I should have. I still hold that I haven't seen any paper where the weight decoupling shows worse performance than the non-decoupled So my current view:
|
@dvolgyes Thanks a lot. It seems weird that the "source code" page for Adam in PyTorch 1,6 is
Then it's not decoupled weight decay. Quite weird, I guess the document page is wrong. But thanks for suggestion, the default weight_decouple is turned on. |
"Code never lies, comments sometimes do." -- Ron Jeffries |
Hi @juntang-zhuang , super excited to try the new improvements.
I saw that you did not updated the ranger version. Do you plan to add the improvements there too?
The text was updated successfully, but these errors were encountered: