Make KneeLRScheduler compatible PyTorch scheduler format #1
Conversation
|
Mighty thanks! Let me spend a few hours going through this PR. Will approve it as soon as I test it on a few testcases and everything works out. |
| infer the total number of steps in the cycle if a value for total_steps is not provided. | ||
| - `pct_start` (float): The percentage of the total steps spent increasing the learning rate. | ||
| - `pct_explore` (float): The percentage of the total steps spent on explore phase (keeping max the learning rate). | ||
| - `cycle_momentum` (bool): If ``True``, momentum is cycled inversely |
There was a problem hiding this comment.
not sure if we want to provide cyclical momentum . No idea how it will affect the explore phase and the effect it may have on reaching a wider minima
There was a problem hiding this comment.
So, it will be better to set this parameter False by default
|
|
||
|
|
||
| class KneeLRScheduler(_LRScheduler): |
There was a problem hiding this comment.
importing _LRScheduler is a good idea, missed it out initially . Will have to change the args as we dont want users to pass in many parameters. Main parameters ( explore , total steps or total epochs , warmup steps). Remaining can be the same as other schedulers in pytorch.
There was a problem hiding this comment.
Actually, there are only 3-4 required args. The rest of args provide customizability of schedule
| class KneeLRScheduler(_LRScheduler): | ||
| """ | ||
| `Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule` |
There was a problem hiding this comment.
Thanks for citing the work
| """ | ||
|
|
||
| def __init__(self, optimizer: Optimizer, max_lr: Union[List[float], float], total_steps: int = None, |
There was a problem hiding this comment.
thanks for importing and using typing. Helps with documentation and readability
|
Heard of you from the issue on PyTorch. Good job! |
|
Thank you! |
KneeLRScheduler now inherits _LRScheduler, supports momentum for Adam and SGD. Add non-zero start lr and non-zero final lr. Add different ways to set up a number of training steps as it is done in OneCycleLR.
Add documentation for the class.