Raise warning for schedulers following chainable shedulers

[vincentqb]

Fixes #29697.

Raise warning for schedulers following chainable schedulers in #26423. See explanation for

• new warning when load/save
• change from deprecation to user warning.

@gchanan -- This should go in the upcoming release following #26423.

Example

Version 1.3.1:

import torch
from torch.nn import Parameter
from torch.optim import SGD
from torch.optim.lr_scheduler import StepLR

model = [Parameter(torch.randn(2, 2, requires_grad=True))]

print('Initial training')

optimizer = SGD(model, 0.1)
scheduler = StepLR(optimizer, 2)

for epoch in range(5):
    print(epoch, optimizer.param_groups[0]['lr'])
    optimizer.step()
    scheduler.step()

state_dict_lr = scheduler.state_dict()

print('Loading from checkpoint')

optimizer = SGD(model, 0.1)
scheduler = StepLR(optimizer, 2)

scheduler.load_state_dict(state_dict_lr)

for epoch in range(5, 10):
    print(epoch, optimizer.param_groups[0]['lr'])
    optimizer.step()
    scheduler.step()

Initial training
0 0.1
1 0.1
2 0.010000000000000002
3 0.010000000000000002
4 0.0010000000000000002
Loading from checkpoint
5 0.1
6 0.00010000000000000003
7 0.00010000000000000003
8 1.0000000000000003e-05
9 1.0000000000000003e-05

This is not correct, but no warnings are raised.

Version 1.3.1 and 1.4.0:

import torch
from torch.nn import Parameter
from torch.optim import SGD
from torch.optim.lr_scheduler import StepLR

model = [Parameter(torch.randn(2, 2, requires_grad=True))]

print('Initial training')

optimizer = SGD(model, 0.1)
scheduler = StepLR(optimizer, 2)

for epoch in range(5):
    print(epoch, optimizer.param_groups[0]['lr'])
    optimizer.step()
    scheduler.step()

state_dict_lr = scheduler.state_dict()
state_dict_opt = optimizer.state_dict()

print('Loading from checkpoint')

optimizer = SGD(model, 0.1)
scheduler = StepLR(optimizer, 2)

scheduler.load_state_dict(state_dict_lr)
optimizer.load_state_dict(state_dict_opt)

for epoch in range(5, 10):
    print(epoch, optimizer.param_groups[0]['lr'])
    optimizer.step()
    scheduler.step()

Initial training
0 lr [0.1]
1 lr [0.1]
2 lr [0.010000000000000002]
3 lr [0.010000000000000002]
4 lr [0.0010000000000000002]
Loading from checkpoint
5 lr [0.0010000000000000002]
6 lr [0.00010000000000000003]
7 lr [0.00010000000000000003]
8 lr [1.0000000000000004e-05]
9 lr [1.0000000000000004e-05]

[facebook-github-bot]

@vincentqb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@vincentqb merged this pull request in 9459db8.

[jrmcornish]

This change seems to raise a warning under all circumstances that state_dict() is called for LambdaLR schedulers, i.e. regardless of whether or not the state of the optimizer is being saved or not. Is this correct? Are we supposed to simply manually suppress this warning if we are using state_dict() correctly? Or are we not supposed to be using state_dict() at all (at least for LambdaLR schedulers)?

[vincentqb]

This change seems to raise a warning under all circumstances that state_dict() is called for LambdaLR schedulers, i.e. regardless of whether or not the state of the optimizer is being saved or not. Is this correct? Are we supposed to simply manually suppress this warning if we are using state_dict() correctly? Or are we not supposed to be using state_dict() at all (at least for LambdaLR schedulers)?

Thanks for reporting this! Now that the warning has been part of 1.4, 1.5, and 1.6, we can remove it and replace it by a message in the documentation. Do you want to open a pull request to do this @jrmcornish ?

Created #43352 to keep track of this.