Warm restart policy is available now

[AutuanLiu]

• Please tell me if violated any rules.

[ssnl]

Can you also add tests for this in test_optim.py?

[ezyang]

@pytorchbot test this please

[AutuanLiu]

ok

[ezyang]

@pytorchbot test this please

[ezyang]

@pytorchbot test this please

[ezyang]

@pytorchbot test this please

[ssnl]

@pytorchbot add to whitelist

[ezyang]

@ssnl Do you think this is OK to merge now?

[apaszke]

@ezyang it's not. See our discussion above, which hasn't concluded yet.

[yf225]

@AutuanLiu Let us know if you have time to address @apaszke and @ssnl 's comments, thanks!

[AutuanLiu]

@yf225 I'm so sorry, I have no time to address and figure out these comments.

[loshchil]

Sorry for not providing the actual working code, but our implementation of restarts for TensorFlow might be useful as a reference:
https://github.com/tensorflow/tensorflow/blob/25c197e02393bd44f50079945409009dd4d434f8/tensorflow/python/training/learning_rate_decay.py#L514

[AlexMRuch]

Looking forward to seeing this function!

[danieltudosiu]

I gave it a shot at porting the TF one, but I am not sure if it is correct since I just started using PyTorch. Could you please give me your two cents?

import math
import warnings

from torch.optim.lr_scheduler import _LRScheduler


class CosineDecayRestarts(_LRScheduler):
    def __init__(
        self,
        optimizer,
        first_decay_steps,
        t_mul=2.0,
        m_mul=1.0,
        alpha=0.0,
        last_epoch=-1,
    ):
        self.first_decay_steps = first_decay_steps
        self.t_mul = t_mul
        self.m_mul = m_mul
        self.alpha = alpha

        super(CosineDecayRestarts, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        if not self._get_lr_called_within_step:
            warnings.warn("To get the last learning rate computed by the scheduler, "
                          "please use `get_last_lr()`.", DeprecationWarning)

        if self.last_epoch == 0:
            return self.base_lrs

        return [self._calculate_decayed_lr(group['lr']) for group in self.optimizer.param_groups]

    def _calculate_decayed_lr(self, group_lr):
        completed_fraction = self._step_count / self.first_decay_steps

        if not self.t_mul == 1.0:
            i_restart = math.floor(
                math.log(1 - completed_fraction * (1 - self.t_mul)) / math.log(self.t_mul)
            )
            sum_r = (1.0 - self.t_mul ** i_restart) / (1.0 - self.t_mul)
            completed_fraction = (completed_fraction - sum_r) / self.t_mul ** i_restart
        else:
            i_restart = math.floor(completed_fraction)
            completed_fraction = completed_fraction - i_restart

        m_fac = self.m_mul ** i_restart
        cosine_decayed = 0.5 * m_fac * (1.0 + math.cos(math.pi * completed_fraction))
        decayed = (1 - self.alpha) * cosine_decayed + self.alpha

        return group_lr * decayed

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
Stale pull requests will automatically be closed 30 days after being marked Stale

[facebook-github-bot]

Hi @AutuanLiu!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.