You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using distributed or parallel set-up in script?: False
Expected behavior
I would expect num_iterations to be clearly documented, and to have a default value.
Additional context
I was originally going to fix this myself, and create a pull request. However, looking into the Ranger21 code, I don't actually know how "num_iterations" was intended to be used? There are two other undocumented arguments: "num_warm_up_iterations" and "num_warm_down_iterations". I don't understand why this wasn't left to the scheduler, and indeed, whether I use "1" num_iterations or the value of my run's total steps - the model does not learn at all. Loss remains flat, weights do not update.
This is not a major issue and I'm going to use a different optimizer for now. Just wanted to make sure maintainers were aware.
The text was updated successfully, but these errors were encountered:
how "num_iterations", "num_warm_up_iterations", "num_warm_down_iterations" was intended to be used?
It's because the Ranger21 optimizer schedules the learning rate with its own recipes (schedulers). Here's the source code Ranger21.
Ranger21 performs linear lr warmup and Explore-exploit lr scheduler.
I don't understand why this wasn't left to the scheduler
I agree with you. It'd be better to use lr scheduler that we want (not hard-coded in the optimizer).
If there is time, I'll try to refactor to remove the internal lr schedulers and implement them with the Pytorch compatible lr scheduler class or something else!
anyway, thanks for the reporting and the suggestion.
Describe the bug
If you try to use the Ranger21 optimizer (with default settings), you'll get an error:
"num_iterations" is an undocumented argument.
To Reproduce
transformers
version: 4.35.2Expected behavior
I would expect num_iterations to be clearly documented, and to have a default value.
Additional context
I was originally going to fix this myself, and create a pull request. However, looking into the Ranger21 code, I don't actually know how "num_iterations" was intended to be used? There are two other undocumented arguments: "num_warm_up_iterations" and "num_warm_down_iterations". I don't understand why this wasn't left to the scheduler, and indeed, whether I use "1" num_iterations or the value of my run's total steps - the model does not learn at all. Loss remains flat, weights do not update.
This is not a major issue and I'm going to use a different optimizer for now. Just wanted to make sure maintainers were aware.
The text was updated successfully, but these errors were encountered: