New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experiment using RAdam optimizer #216

Open

LifeIsStrange opened this issue Aug 18, 2019 · 1 comment

LifeIsStrange commented Aug 18, 2019 •

edited

Loading

Instead of adam.
https://arxiv.org/pdf/1908.03265v1

Lookahead merits to be tried too https://arxiv.org/pdf/1907.08610v1.pdf
Maybe it can be used on top of RAdam.

LifeIsStrange mentioned this issue

Experiment using state of the art activation functions #219

Open

Author

LifeIsStrange commented Aug 29, 2019 •

edited

Loading

"Maybe it can be used on top of RAdam"
My intuition turned out to be right!

There is now a ready made combination of both Optimizers:
https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d
This is the one you should try, Ranger.

https://forums.fast.ai/t/meet-ranger-radam-lookahead-optimizer/52886

Some insights:
https://forums.fast.ai/t/how-we-beat-the-5-epoch-imagewoof-leaderboard-score-some-new-techniques-to-consider/53453

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment