Inconsistant values of lr_scheduler.get_lr and lr in optimizer.param_groups

[lihuanglx]

🐛 Bug

After upgrading to 1.1.0, the value returned by lr_scheduler.get_lr is confusing comparing to the lr value inside optimizer.param_groups.

To Reproduce

Here I follow the new convention putting the lr_scheduler.step() at the end of each iteration, see the new documents and #7889 (which is probably the root of this issue).

Code:

# 1.1.0
import torch

net = torch.nn.Conv2d(1, 1, 1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)

for i in range(5):
    print(i, lr_scheduler.get_lr(), optimizer.param_groups[0]['lr'])
    lr_scheduler.step()

Output:

0 [0.1] 0.1
1 [0.08100000000000002] 0.09000000000000001
2 [0.07290000000000002] 0.08100000000000002
3 [0.06561000000000002] 0.07290000000000002
4 [0.05904900000000002] 0.06561000000000002

We got inconsistant values. The two values are the same in the first line, but different in following lines by lr decay factor gamma = 0.9.

Expected behavior

At least we should have consistant values of the two, right?

In the old version 1.0.1, if we follow the previous convention putting the lr_scheduler.step() at the beginning of each iteration, the output values are reasonable and consistant:

Code:

# 1.0.1
import torch

net = torch.nn.Conv2d(1, 1, 1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)

for i in range(5):
    lr_scheduler.step()
    print(i, lr_scheduler.get_lr(), optimizer.param_groups[0]['lr'])

Output:

0 [0.1] 0.1
1 [0.09000000000000001] 0.09000000000000001
2 [0.08100000000000002] 0.08100000000000002
3 [0.0729] 0.0729
4 [0.06561] 0.06561

Environment

I'll skip this part since it can be easily reproduced from a fresh 1.1.0 installation.

[codexetreme]

If this is a bug, can I try to work on it?

[vfdev-5]

@lihuanglx I would say this is also due to recursive reimplementation of LR schedulers, #14010
such that, multiple schedulers can be used simultaneously to modify the learning rates.