Learning rate initialization of create_lr_scheduler_with_warmup

[developer0hye]

🐛 Bug description

It seems that learning rate initialization for create_lr_scheduler_with_warmup is not working well.

how to reproduce it

import torchvision.models as models
import torch.optim as optim
from ignite.handlers.param_scheduler import create_lr_scheduler_with_warmup 

model = models.resnet18()

total_iteration = 100
warmup_iteration = 10
initial_lr = 1e-3
warmup_initial_lr = 1e-5

optimizer = optim.Adam(model.parameters(), lr=initial_lr)
lr_scheduler = create_lr_scheduler_with_warmup(optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=total_iteration),
                                               warmup_start_value=warmup_initial_lr,
                                               warmup_duration=warmup_iteration,
                                               warmup_end_value=initial_lr)

for _ in range(total_iteration):
  print(optimizer.param_groups[0]['lr'])
  lr_scheduler(None)

Results

0.001
1e-05
0.00012
0.00023
...
2.9559615522887284e-05

I thought that learning rate of optimizer should have been 1e-5. But I got 1e-3.

Environment

• PyTorch Version (e.g., 1.4):
• Ignite Version (e.g., 0.3.0):
• OS (e.g., Linux):
• How you installed Ignite (conda, pip, source): pip
• Python version:
• Any other relevant information: Run on Colab

[vfdev-5]

@developer0hye thanks for reporting. Actually it depends on when you check the learning rate: before / after applying the scheduler.

Learning rate value is applied to the optimizer when lr_scheduler(None). Here is master docs with fixed example: https://pytorch.org/ignite/master/generated/ignite.handlers.param_scheduler.create_lr_scheduler_with_warmup.html (v0.4.8 doc's example is wrong: Events.ITERATION_COMLETED -> Events.ITERATION_STARTED).
Here is your code with more prints to show how it works.

import torchvision.models as models
import torch.optim as optim
from ignite.handlers.param_scheduler import create_lr_scheduler_with_warmup 

model = models.resnet18()

total_iteration = 20
warmup_iteration = 10
initial_lr = 1e-3
warmup_initial_lr = 1e-5

optimizer = optim.Adam(model.parameters(), lr=initial_lr)
lr_scheduler = create_lr_scheduler_with_warmup(optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=total_iteration),
                                               warmup_start_value=warmup_initial_lr,
                                               warmup_duration=warmup_iteration,
                                               warmup_end_value=initial_lr)

print("Optimizer's lr: ", optimizer.param_groups[0]['lr'])
for i in range(total_iteration):
    print(i, "previous lr: ", optimizer.param_groups[0]['lr'], end=" -> ")
    lr_scheduler(None)
    print("applied lr: ", optimizer.param_groups[0]['lr'])

Optimizer's lr:  0.001
0 previous lr:  0.001 -> applied lr:  1e-05
1 previous lr:  1e-05 -> applied lr:  0.00012
2 previous lr:  0.00012 -> applied lr:  0.00023
3 previous lr:  0.00023 -> applied lr:  0.00034
4 previous lr:  0.00034 -> applied lr:  0.00045000000000000004
5 previous lr:  0.00045000000000000004 -> applied lr:  0.00056
6 previous lr:  0.00056 -> applied lr:  0.00067
7 previous lr:  0.00067 -> applied lr:  0.0007800000000000001
8 previous lr:  0.0007800000000000001 -> applied lr:  0.0008900000000000001
9 previous lr:  0.0008900000000000001 -> applied lr:  0.001
10 previous lr:  0.001 -> applied lr:  0.0009938441702975688
11 previous lr:  0.0009938441702975688 -> applied lr:  0.0009755282581475768
12 previous lr:  0.0009755282581475768 -> applied lr:  0.0009455032620941839
13 previous lr:  0.0009455032620941839 -> applied lr:  0.0009045084971874737
14 previous lr:  0.0009045084971874737 -> applied lr:  0.0008535533905932737
15 previous lr:  0.0008535533905932737 -> applied lr:  0.0007938926261462366
16 previous lr:  0.0007938926261462366 -> applied lr:  0.0007269952498697733
17 previous lr:  0.0007269952498697733 -> applied lr:  0.0006545084971874737
18 previous lr:  0.0006545084971874737 -> applied lr:  0.0005782172325201155
19 previous lr:  0.0005782172325201155 -> applied lr:  0.0005

[developer0hye]

@vfdev-5
Thanks for your reply!

Should I call lr_scheduler(None) to train model with warmup_start_value at first iteration before optimizer.step()?

for _ in range(iteration):
 ...
 lr_scheduler(None)
 ...
 optimizer.step()
 ...
```python 
 

[vfdev-5]

@developer0hye yes, we can call it manually like that or attach to the engine for Events.ITERATION_STARTED:

from torch.optim.lr_scheduler import ExponentialLR

scheduler = create_lr_scheduler_with_warmup(torch_lr_scheduler,
                                            warmup_start_value=0.0,
                                            warmup_end_value=0.1,
                                            warmup_duration=3)

def train_step(engine, batch):
    ...
    optimizer.step()

trainer = Engine(train_step)

# scheduler(trainer) will be called before calling `train_step`
trainer.add_event_handler(Events.ITERATION_STARTED, scheduler)

[developer0hye]

@vfdev-5
Thanks!

[developer0hye]

@vfdev-5

I made a repository related to learning rate warmup method on Torch with pytorch-ignite!

https://github.com/developer0hye/Torch-Warmup

[vfdev-5]

@developer0hye Sounds great ! By the way, using ignite's param schedulers in addition to LR scheduling you can also update other params like momentum (SGD), betas (Adam) etc.

[developer0hye]

@vfdev-5
Wow! I have to study pytorch-ignite harder! Thanks for sharing your works!

[vfdev-5]

@vfdev-5 Wow! I have to study pytorch-ignite harder! Thanks for sharing your works!

It is a joint work of many contributors, thanks to all of them!
By the way, if you are interested in helping the project, feel free to connect to our discord server: https://pytorch-ignite.ai/chat and join #start-contributing channel.

[developer0hye]

@vfdev-5
I joined! Thanks!