Add cyclical learning rate schedulers #644

RaphaelMeudec · 2019-10-29T23:07:53Z

Migrating CyclicLR from keras-team/keras-contrib as discussed in keras-team/keras-contrib#519.

@gabrieldemarmiesse & @seanpmorgan would love some feedback on implementation
@seanpmorgan is there any recommendations regarding tests? I plan on taking inspiration from tf core schedulers, does it seem ok to you?

googlebot · 2019-10-29T23:07:58Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

RaphaelMeudec · 2019-10-29T23:08:49Z

@googlebot I signed it!

googlebot · 2019-10-29T23:08:51Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

gabrieldemarmiesse · 2019-10-30T09:18:43Z

I'm surprised. Do you guys have a continuous integration service running for pull requests?

gabrieldemarmiesse

Maybe some tests would help ensure that no bugs are introduced?

RaphaelMeudec · 2019-10-30T13:47:06Z

I'm surprised. Do you guys have a continuous integration service running for pull requests?

Seems that CI is triggered by adding the kokoro:force-run tag

Maybe some tests would help ensure that no bugs are introduced?

Of course! Wanted to check first that implementation I used seems ok.

gabrieldemarmiesse · 2019-10-30T13:50:31Z

Looks good to me :)

seanpmorgan · 2019-10-30T14:42:11Z

Migrating CyclicLR from keras-team/keras-contrib as discussed in keras-team/keras-contrib#519.

@gabrieldemarmiesse & @seanpmorgan would love some feedback on implementation

@seanpmorgan is there any recommendations regarding tests? I plan on taking inspiration from tf core schedulers, does it seem ok to you?

Thanks @RaphaelMeudec implementation looks great. As far as test cases go, matching as many tests as you can from TF core schedulers is ideal.

We had a pretty nasty build break this morning... I will be looking into it shortly, but if you would like to run tests in the mean time just pin the nightly to yesterday's version:
https://github.com/tensorflow/addons/blob/master/build_deps/requirements.txt

If it doesn't seem there is a quick fix then we'll pin the master branch as well.

RaphaelMeudec · 2019-11-04T20:16:29Z

@seanpmorgan Added the tests, ready for review

seanpmorgan · 2019-11-05T02:18:17Z

Thanks @RaphaelMeudec! Looks like there are some line too long lint errors. Could you run this to auto format the code
bash tools/run_docker.sh -c 'make code-format'

RaphaelMeudec · 2019-11-05T03:52:17Z

@seanpmorgan This is weird, the code-format is not doing any change. Is there a difference between the code-format and sanity-check? I'll fix the errors tomorrow

WindQAQ · 2019-11-05T05:15:59Z

@seanpmorgan This is weird, the code-format is not doing any change. Is there a difference between the code-format and sanity-check? I'll fix the errors tomorrow

yapf does not aggressively make the length of lines fit 80 characters so you should manually modify it in most of the time.

RaphaelMeudec · 2019-11-05T15:27:41Z

@WindQAQ Fixed the errors, should be good now

RaphaelMeudec · 2019-11-06T14:18:31Z

@seanpmorgan Updated to register_keras_serializable, could you rerun kokoro?

seanpmorgan

Implementation looks great. Small nit and then would you mind updating the TFA Optimizers README:
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/README.md

tensorflow_addons/optimizers/cyclical_learning_rate.py

seanpmorgan

Thanks! One last nit

seanpmorgan · 2019-11-08T21:32:06Z

tensorflow_addons/optimizers/README.md

@@ -10,6 +10,7 @@
 | moving_average | Dheeraj R. Reddy | dheeraj98reddy@gmail.com |
 | rectified_adam | Zhao Hanguang | cyberzhg@gmail.com |
 | weight_decay_optimizers |  Phil Jund | ijund.phil@googlemail.com   |
+| cyclical_learning_rate | Raphael Meudec | raphael.meudec@gmail.com |


Alphabetize please

What do you mean by alphabetize?

Just order alphabetically:

i.e. place it below conditional gradient in the list

seanpmorgan · 2019-11-08T21:32:22Z

tensorflow_addons/optimizers/README.md

@@ -22,6 +23,7 @@
 | moving_average | MovingAverage | |
 | rectified_adam | RectifiedAdam | https://arxiv.org/pdf/1908.03265v1.pdf |
 | weight_decay_optimizers | SGDW, AdamW, extend_with_decoupled_weight_decay | https://arxiv.org/pdf/1711.05101.pdf |
+| cyclical_learning_rate | Cyclical Learning Rate | https://arxiv.org/abs/1506.01186 |


Alphabetize please

RaphaelMeudec · 2019-11-12T08:15:51Z

@seanpmorgan Alphabetized it and rebased on tensorflow/addons. You might want to re-run kokoro

seanpmorgan

LGTM thanks for the contribution!

gabrieldemarmiesse · 2019-11-12T15:56:51Z

Thanks a lot for the contribution @RaphaelMeudec and thank you @seanpmorgan for the review!

Hecim1984

🚀

RaphaelMeudec requested review from facaiy, Squadrick and WindQAQ as code owners October 29, 2019 23:07

googlebot added the cla: no label Oct 29, 2019

googlebot added cla: yes and removed cla: no labels Oct 29, 2019

gabrieldemarmiesse reviewed Oct 30, 2019

View reviewed changes

seanpmorgan added the kokoro:force-run label Oct 30, 2019

kokoro-team removed the kokoro:force-run label Oct 30, 2019

seanpmorgan added the optimizers label Oct 31, 2019

RaphaelMeudec changed the title ~~[wip] Add cyclical learning rate schedulers~~ Add cyclical learning rate schedulers Nov 4, 2019

seanpmorgan added the kokoro:force-run label Nov 5, 2019

kokoro-team removed the kokoro:force-run label Nov 5, 2019

seanpmorgan added the kokoro:force-run label Nov 6, 2019

kokoro-team removed the kokoro:force-run label Nov 6, 2019

seanpmorgan added the kokoro:force-run label Nov 6, 2019

kokoro-team removed the kokoro:force-run label Nov 6, 2019

seanpmorgan self-assigned this Nov 6, 2019

seanpmorgan requested changes Nov 8, 2019

View reviewed changes

tensorflow_addons/optimizers/cyclical_learning_rate.py Outdated Show resolved Hide resolved

tensorflow_addons/optimizers/cyclical_learning_rate.py Outdated Show resolved Hide resolved

tensorflow_addons/optimizers/cyclical_learning_rate.py Outdated Show resolved Hide resolved

seanpmorgan requested changes Nov 8, 2019

View reviewed changes

Raphael Meudec added 10 commits November 12, 2019 09:07

Add cyclical learning rate schedulers

47c64e1

Apply code formatting

34359c5

Add test file

e14ea41

Start working on the tests

a96190f

Format code

ba41500

Last test for custom cyclical lr

52234ce

Fix linting errors

83404e0

Move to register_keras_serializable

ada154f

Review comments

da1679b

alphabetic order

2ed2075

seanpmorgan added the kokoro:force-run label Nov 12, 2019

kokoro-team removed the kokoro:force-run label Nov 12, 2019

seanpmorgan approved these changes Nov 12, 2019

View reviewed changes

seanpmorgan merged commit 0a225c7 into tensorflow:master Nov 12, 2019

Hecim1984 reviewed Sep 28, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cyclical learning rate schedulers #644

Add cyclical learning rate schedulers #644

RaphaelMeudec commented Oct 29, 2019

googlebot commented Oct 29, 2019

RaphaelMeudec commented Oct 29, 2019

googlebot commented Oct 29, 2019

gabrieldemarmiesse commented Oct 30, 2019

gabrieldemarmiesse left a comment

RaphaelMeudec commented Oct 30, 2019

gabrieldemarmiesse commented Oct 30, 2019

seanpmorgan commented Oct 30, 2019

RaphaelMeudec commented Nov 4, 2019

seanpmorgan commented Nov 5, 2019 •

edited

Loading

RaphaelMeudec commented Nov 5, 2019

WindQAQ commented Nov 5, 2019

RaphaelMeudec commented Nov 5, 2019

RaphaelMeudec commented Nov 6, 2019

seanpmorgan left a comment

seanpmorgan left a comment

seanpmorgan Nov 8, 2019

RaphaelMeudec Nov 8, 2019

seanpmorgan Nov 8, 2019

seanpmorgan Nov 8, 2019

RaphaelMeudec Nov 9, 2019

seanpmorgan Nov 8, 2019

RaphaelMeudec commented Nov 12, 2019

seanpmorgan left a comment

gabrieldemarmiesse commented Nov 12, 2019

Hecim1984 left a comment

Add cyclical learning rate schedulers #644

Add cyclical learning rate schedulers #644

Conversation

RaphaelMeudec commented Oct 29, 2019

googlebot commented Oct 29, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

RaphaelMeudec commented Oct 29, 2019

googlebot commented Oct 29, 2019

gabrieldemarmiesse commented Oct 30, 2019

gabrieldemarmiesse left a comment

Choose a reason for hiding this comment

RaphaelMeudec commented Oct 30, 2019

gabrieldemarmiesse commented Oct 30, 2019

seanpmorgan commented Oct 30, 2019

RaphaelMeudec commented Nov 4, 2019

seanpmorgan commented Nov 5, 2019 • edited Loading

RaphaelMeudec commented Nov 5, 2019

WindQAQ commented Nov 5, 2019

RaphaelMeudec commented Nov 5, 2019

RaphaelMeudec commented Nov 6, 2019

seanpmorgan left a comment

Choose a reason for hiding this comment

seanpmorgan left a comment

Choose a reason for hiding this comment

seanpmorgan Nov 8, 2019

Choose a reason for hiding this comment

RaphaelMeudec Nov 8, 2019

Choose a reason for hiding this comment

seanpmorgan Nov 8, 2019

Choose a reason for hiding this comment

seanpmorgan Nov 8, 2019

Choose a reason for hiding this comment

RaphaelMeudec Nov 9, 2019

Choose a reason for hiding this comment

seanpmorgan Nov 8, 2019

Choose a reason for hiding this comment

RaphaelMeudec commented Nov 12, 2019

seanpmorgan left a comment

Choose a reason for hiding this comment

gabrieldemarmiesse commented Nov 12, 2019

Hecim1984 left a comment

Choose a reason for hiding this comment

seanpmorgan commented Nov 5, 2019 •

edited

Loading