-
Notifications
You must be signed in to change notification settings - Fork 45.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding LARS to ResNet #6327
Adding LARS to ResNet #6327
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a couple minor comments, but overall this looks good. Thanks!
for batch size). The learning rate is then decayed using a polynomial rate | ||
decay schedule with power 2.0. | ||
|
||
Args: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where are these args?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.
official/resnet/imagenet_main.py
Outdated
@@ -44,6 +44,19 @@ | |||
|
|||
DATASET_NAME = 'ImageNet' | |||
|
|||
flags.DEFINE_bool('enable_lars', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you move these into define_resnet_flags
in resnet_run_loop.py
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When you are finished with all the comments make sure you fix the lint errors we cleaned them all up and do not want to add new ones. You can see the errors by clicking the link that will take you to sponge. The log is hard to read but it tells you lines,files, and reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM from me. Lint looks ok and the very exact settings per batch are ok for now.
returns the current learning rate | ||
""" | ||
|
||
# Learning rate schedule for LARS polynomial schedule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is different from before. Shouldn't it be:
if flags.FLAGS.batch_size < 8192:
plr = 5.0
w_epochs = 5
elif flags.FLAGS.batch_size < 16384:
plr = 10.0
w_epochs = 5
if flags.FLAGS.batch_size < 32768:
plr = 25.0
w_epochs = 5
else:
plr = 32.0
w_epochs = 14
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
facepalm. fixed.
Thanks! Enjoy the big shiny green button. |
* Adding LARS to ResNet * Fixes for the LARS patch * Fixes for the LARS patch * more fixes * 1 more fix
Adding LARS to ResNet