[Feature Request] Cyclical Learning Rates

[Kaixhin]

Cyclical learning rates seem to be becoming more popular. Given that we now have a lr scheduler, it would be nice to have this available as another schedule scheme. Pinging @ajbrock, who has experience with this (unlike me haha).

[ajbrock]

I'm down, though I wasn't aware of the LR scheduler API so I'll need to dig into that and learn it before I can contribute. I personally have seen more uses of SGDR style learning rates than cyclic learning rates (the only two cites on the CLR paper are self-cites) but I don't think either should be too tough to throw together.

[Jiaming-Liu]

Hi. I am the puller of LR Scheduler. IMO, you should be able to easily implement this using class LambdaLR. @ajbrock