-
Notifications
You must be signed in to change notification settings - Fork 611
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cyclical learning rate schedulers #644
Add cyclical learning rate schedulers #644
Conversation
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
@googlebot I signed it! |
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
I'm surprised. Do you guys have a continuous integration service running for pull requests? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe some tests would help ensure that no bugs are introduced?
Seems that CI is triggered by adding the kokoro:force-run tag
Of course! Wanted to check first that implementation I used seems ok. |
Looks good to me :) |
Thanks @RaphaelMeudec implementation looks great. As far as test cases go, matching as many tests as you can from TF core schedulers is ideal. We had a pretty nasty build break this morning... I will be looking into it shortly, but if you would like to run tests in the mean time just pin the nightly to yesterday's version: If it doesn't seem there is a quick fix then we'll pin the master branch as well. |
@seanpmorgan Added the tests, ready for review |
Thanks @RaphaelMeudec! Looks like there are some line too long lint errors. Could you run this to auto format the code |
@seanpmorgan This is weird, the |
|
@WindQAQ Fixed the errors, should be good now |
@seanpmorgan Updated to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implementation looks great. Small nit and then would you mind updating the TFA Optimizers README:
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/README.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! One last nit
@@ -10,6 +10,7 @@ | |||
| moving_average | Dheeraj R. Reddy | dheeraj98reddy@gmail.com | | |||
| rectified_adam | Zhao Hanguang | cyberzhg@gmail.com | | |||
| weight_decay_optimizers | Phil Jund | ijund.phil@googlemail.com | | |||
| cyclical_learning_rate | Raphael Meudec | raphael.meudec@gmail.com | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alphabetize please
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean by alphabetize?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just order alphabetically:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i.e. place it below conditional gradient in the list
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
@@ -22,6 +23,7 @@ | |||
| moving_average | MovingAverage | | | |||
| rectified_adam | RectifiedAdam | https://arxiv.org/pdf/1908.03265v1.pdf | | |||
| weight_decay_optimizers | SGDW, AdamW, extend_with_decoupled_weight_decay | https://arxiv.org/pdf/1711.05101.pdf | | |||
| cyclical_learning_rate | Cyclical Learning Rate | https://arxiv.org/abs/1506.01186 | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alphabetize please
@seanpmorgan Alphabetized it and rebased on tensorflow/addons. You might want to re-run kokoro |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM thanks for the contribution!
Thanks a lot for the contribution @RaphaelMeudec and thank you @seanpmorgan for the review! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Migrating CyclicLR from keras-team/keras-contrib as discussed in keras-team/keras-contrib#519.