Replies: 2 comments 2 replies
-
FYI - I am in the process of documenting the various defaults and available schedulers as part of timmdocs. Recently, I just added some more documentation for the available Optimizers too - https://fastai.github.io/timmdocs/Optimizers. :) |
Beta Was this translation helpful? Give feedback.
1 reply
-
Hi @ross, it is great news to see timmdocs.
The documents in both timmdocs and fastai are explained concisely and precisely.
I love reading them.
Best regards
Linh
… On 7 Mar 2021, at 07:36, Aman Arora ***@***.***> wrote:
FYI - I'll am in the process of documenting the various defaults and available schedulers as part of timmdocs.
Recently, I just added some more documentation for the available Optimizers too - https://fastai.github.io/timmdocs/Optimizers. :)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hey Ross! Hope you're well.
I have a quick question regarding the LR schedulers especially the "cosine" scheduler.
From what I can see in the
args
defaults in the training script, we set:That is
lr_cycle_limit
is set to 1. Now as per my understanding, in SGDR we restart the learning rate after some epochs so that the LR schedule looks something like:However, since the defaults in training script set
lr_cycle_limit
to 1, we never actually restart the LR schedule. Therefore, based on the implementation, the LR schedule ends up looking something like:I am wondering if we should update the defaults or would it make sense to have a Callback that tracks the learning rate similar to
fastai
and plots the LR for more visibility of the various schedulers.I have shared my implementation in this nb as gist - https://gist.github.com/amaarora/8f328bb19d2d262e5d172601d00fa330.
Keen to hear your thoughts and understand whether my understanding of schedulers in
timm
is correct? Thanks! :)Beta Was this translation helpful? Give feedback.
All reactions