CosineAnnealingWarmRestarts is not compatible with LRScheduler.simulate_values()

[sdesrozis]

🐛 Bug description

CosineAnnealingWarmRestarts is not compatible with LRScheduler.simulate_values(). Precisely, a object of type CosineAnnealingWarmRestarts can't be replicated by LRScheduler._replicate_lr_scheduler(). It works for CosineAnnealingLR.

For example :

lr_scheduler = CosineAnnealingWarmRestarts(optimizer=optimizer, T_0=10, eta_min=1e-3)
lr_values = LRScheduler.simulate_values(num_events=50, lr_scheduler=lr_scheduler)

produces the following error

Traceback (most recent call last):
  File "tutorials/misc/lr_schedulers.py", line 56, in <module>
    lr_values = LRScheduler.simulate_values(num_events=50, lr_scheduler=lr_scheduler)
  File "/work/desrozis/Softwares/conda/envs/focus-light/lib/python3.7/site-packages/ignite/contrib/handlers/param_scheduler.py", line 606, in simulate_values
    copy_lr_scheduler = LRScheduler._replicate_lr_scheduler(lr_scheduler)
  File "/work/desrozis/Softwares/conda/envs/focus-light/lib/python3.7/site-packages/ignite/contrib/handlers/param_scheduler.py", line 627, in _replicate_lr_scheduler
    copy_lr_scheduler = lr_scheduler_cls(optimizer=dummy_optimizer, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'T_i'

This issue is due to the assumption that lr_scheduler.__state_dict__ contains arguments of method CosineAnnealingWarmRestarts.__init__(). A workaround could be to remove T_i in a similar way to base_lrs and last_epoch but it's not very satisfactory...

Hope it helps to have a better and stonger ignite.

Environment

• PyTorch Version (e.g., 1.4): 1.4
• Ignite Version (e.g., 0.3.0): 0.3.0
• OS (e.g., Linux): Linux
• How you installed Ignite (conda, pip, source): conda
• Python version: 3.7.6
• Any other relevant information:

[vfdev-5]

Thanks for the report @sdesrozis ! I'll investigate the issue and push a fix.

EDIT: Seems like implementation of CosineAnnealingWarmRestarts is a bit different from other LR schedulers, e.g. CosineAnnealingLR. This implies that it wont work even for LR scheduling (not simulation of lr values). The difference is in step and get_lr implementations. In CosineAnnealingWarmRestarts, get_lr does not account on optimizer's lr vs CosineAnnealingLR and others. So, get_lr is stateless but internally in ignite, we use only get_lr without calling step. Finally, this gives a constant learning rate.

[sdesrozis]

What about copy.deepcopy() ? Does it work on a _LRScheduler ? It could replace the home made replication ?

[sdesrozis]

The bug is also with MultiplicativeLR. The way to replicate _LRScheduler is really not safe. This scheduler have an argument lr_lambda but it stores in self.lr_lambdas. To replicate, we use self.__state_dict__ and pass it to __init__. It can't works.
IMO plot values with replication is very tricky.

[sdesrozis]

What about copy.deepcopy() ? Does it work on a _LRScheduler ? It could replace the home made replication ?

copy.deepcopy() will replicate params of the related optimizer, not a good idea.

[sdesrozis]

@vfdev-5 maybe we should accept that we can't use a LRScheduler after ploting values ? (and do not replicate but can plot for every schedulers).

[vfdev-5]

@sdesrozis how about the same approach as used with lr finders etc :

• save initial state dicts
• do something
• restore init state
  ?

[sdesrozis]

You save my day 😊 I check if it’s ok.