Add Cosine Annealing LR scheduler as a decay method

[arnavgarg1]

Implements Cosine Annealing: https://arxiv.org/pdf/1608.03983.pdf

CosineAnnealingWithRestarts is a scheduling technique that starts with a very large learning rate and then aggressively decreases it to a value near 0 before increasing the learning rate again. This happens either at fixed intervals, or are growing intervals, both of which can be configured.

To enable cosine learning rate scheduling with warm restarts, you can set:

trainer:
    learning_rate_scheduler:
        decay: cosine

There are 3 optional parameters that you can configure:

• T_0: The number of steps before warm restarts. Defaults to steps_per_checkpoint if not specified.
• T_mult: Period between warm restarts. Defaults to 1. If set to 2 or greater, it multiplies the current interval by the multiplier between warm restarts. For e.g., if T_0 is 20 steps and T_mult is 2, then the periods between restarts will be 20, 40, 80, 160, ....
• eta_min: The minimum learning rate the LR should be decayed to. Defaults to 0.

This PR also refactors the way the learning rate schedulers are configured, using a new SequentialLR object.

[github-actions]

Unit Test Results

  6 files  ±0    6 suites  ±0   1h 1m 32s ⏱️ - 3m 15s
34 tests ±0  29 ✔️ ±0    5 💤 ±0  0 ❌ ±0 
88 runs  ±0  72 ✔️ ±0  16 💤 ±0  0 ❌ ±0 

Results for commit c297a99. ± Comparison against base commit ccbf5de.

♻️ This comment has been updated with latest results.