lr scheduling

[egozi]

Is your feature request related to a problem? Please describe.
Add more learning rate scheduling options

Describe the solution you'd like
Include a flag/operation to choose other lr scheduling than reduce-on-plateau.
In particular, cosine-annealing scheduling is very popular and should be part of the package.

Describe alternatives you've considered
NA

Additional context
NA

[mosheraboh]

@egozi, you should be able to directly use any implemented PyTorch lr sch in here https://pytorch.org/docs/stable/optim.html - it includes cosine-annealing.
Did you have an issue with it?