Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement RAdam optimizer #422

Closed
seanpmorgan opened this issue Aug 16, 2019 · 17 comments
Closed

Implement RAdam optimizer #422

seanpmorgan opened this issue Aug 16, 2019 · 17 comments

Comments

@seanpmorgan
Copy link
Member

@seanpmorgan seanpmorgan commented Aug 16, 2019

System information

  • Are you willing to contribute it (yes/no): Yes, but would like to see if someone else will
  • Are you willing to maintain it going forward? (yes/no): Yes, but would like to see if someone else will

Describe the feature and the current behavior/state.
New paper describing RAdam looks like a drop in replacement for Adam optimizer with better results.

https://arxiv.org/abs/1908.03265v1
https://github.com/LiyuanLucasLiu/RAdam

@yassineAlouini

This comment has been minimized.

Copy link

@yassineAlouini yassineAlouini commented Aug 16, 2019

That would be awesome indeed.
For the record, here is a Keras implementation (not official): https://github.com/CyberZHG/keras-radam.

@rafiqhasan

This comment has been minimized.

Copy link

@rafiqhasan rafiqhasan commented Aug 17, 2019

This is going to be a great addon ! For any one who is looking out for another detailed comparison can read through - https://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b

@talipini

This comment has been minimized.

Copy link

@talipini talipini commented Aug 19, 2019

I vote for this as well. Looking for this!

@SSaishruthi

This comment has been minimized.

Copy link
Contributor

@SSaishruthi SSaishruthi commented Aug 19, 2019

Great addition.
Can I try implementation of this?
@seanpmorgan

@seanpmorgan

This comment has been minimized.

Copy link
Member Author

@seanpmorgan seanpmorgan commented Aug 19, 2019

@SSaishruthi Sure! I know @sayoojbk has also shared interest in helping with this so if you could make a WIP PR as soon as you get it started that'd be great so we could have a few eyes on this and push it through.

@physicso

This comment has been minimized.

Copy link

@physicso physicso commented Aug 25, 2019

Looking forward to that!

@WindQAQ

This comment has been minimized.

Copy link
Member

@WindQAQ WindQAQ commented Aug 30, 2019

Seems that there is an unofficial implementation for TF/Keras.
https://github.com/CyberZHG/keras-radam

@physicso

This comment has been minimized.

Copy link

@physicso physicso commented Aug 30, 2019

@SSaishruthi

This comment has been minimized.

Copy link
Contributor

@SSaishruthi SSaishruthi commented Aug 30, 2019

Thanks for the links. I have marked all the links and planning to kick start with the implementation after this weekend. Just done with other priority change. Will keep posted.

@luminoso

This comment has been minimized.

Copy link

@luminoso luminoso commented Sep 3, 2019

@SSaishruthi it looks like RAdam has already an improvement called 'RAdam+lookahead'. One possible implementation of lookahead:
https://github.com/bojone/keras_lookahead (been testing this one myself)

It's called 'Ranger' (the combination of RAdam + lookahead):
Small article about it: https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d

Paper: https://arxiv.org/abs/1907.08610v1
Lookahead Pytorch implementation: https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py

@makercob

This comment has been minimized.

Copy link

@makercob makercob commented Sep 4, 2019

@luminoso however, it seems that that repo does not support TensorFlow.

@seanpmorgan

This comment has been minimized.

Copy link
Member Author

@seanpmorgan seanpmorgan commented Sep 10, 2019

Thanks for the links. I have marked all the links and planning to kick start with the implementation after this weekend. Just done with other priority change. Will keep posted.

Hi @SSaishruthi I know you're working on several different things, including the core migration of F1. Would you be okay with @AakashKumarNain taking a look at this one as he has expressed interest. Would love for you to help review any implementation.

@SSaishruthi

This comment has been minimized.

Copy link
Contributor

@SSaishruthi SSaishruthi commented Sep 10, 2019

@seanpmorgan Sure. Will collaborate along so that we keep things going.

@AakashKumarNain

This comment has been minimized.

Copy link
Contributor

@AakashKumarNain AakashKumarNain commented Sep 10, 2019

@seanpmorgan @SSaishruthi. The keras implementation as pointed out in the comments LGTM. Also, it is written with Optimizer_v2 api. Take a look

https://github.com/CyberZHG/keras-radam/blob/master/keras_radam/optimizer_v2.py

@seanpmorgan

This comment has been minimized.

Copy link
Member Author

@seanpmorgan seanpmorgan commented Sep 10, 2019

Ping @CyberZHG. Would it be okay to use your implementation as part of Addons? The license you have on it looks like it'd be okay -- but wanted to get your permission/see if you'd like to contribute it yourself?

@AakashKumarNain

This comment has been minimized.

Copy link
Contributor

@AakashKumarNain AakashKumarNain commented Sep 10, 2019

Yeah it is fair if @CyberZHG just adds it here. Most of the work is already done in that.

@CyberZHG

This comment has been minimized.

Copy link
Contributor

@CyberZHG CyberZHG commented Sep 13, 2019

I'll try to migrate the codes and make a PR in the next few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.