about lr_scheduler

[laisimiao]

about lr_scheduler, i have two questions:

• 1. In train.py, lr_scheduler.step(epoch) write in line194 before optimizer.step() in line223, it seems it disobey pytorch rule about writing schedule after optimizer.step(), but it didn't throw error, why?
• 2. When cfg.BACKBONE.TRAIN_EPOCH == epoch, I know it should begin training backbone, but I don't know why now set cfg.TRAIN.START_EPOCH as last_epoch, this will lead lr decay largely when epoch=cfg.BACKBONE.TRAIN_EPOCH +1?

[ZhiyuanChen]

In train.py, lr_scheduler.step(epoch) write in line194 before optimizer.step() in line223, it seems it disobey pytorch rule about writing schedule after optimizer.step(), but it didn't throw error, why?

It is intended for PyTorch 0.4.1, where the call of lr_scheduler.step() should be before optimizer.step()

[laisimiao]

Oh I got it Thank you. But I run this framework in my pytorch1.2 env and it dosen't give any warnings, so I am so curious. Fine, what's explanation about question 2?

[ZhiyuanChen]

Oh I got it Thank you. But I run this framework in my pytorch1.2 env and it dosen't give any warnings, so I am so curious.

Please refer to PyTorch documentation for detailed explanations:
Prior to PyTorch 1.1.0, the learning rate scheduler was expected to be called before the optimizer’s update; 1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. If you are unable to reproduce results after upgrading to PyTorch 1.1.0, please check if you are calling scheduler.step() at the wrong time.

Fine, what's explanation about question 2?
I don't have enough time to look into details now, perheps you can wait for a few days before I can check it for you

[laisimiao]

That's why I ask it.

I don't have enough time to look into details now, perheps you can wait for a few days before I can check it for you

Thank you!

[laisimiao]

Oh I also find a code logic bug:
In codes(train.py), use method build_opt_lr to construct an optimizer and lr_scheduler, so learning rate will adjust by lr_scheduler. But there are also some codes in this method like:

trainable_params = []
    trainable_params += [{'params': filter(lambda x: x.requires_grad,
                                           model.backbone.parameters()),
                          'lr': cfg.BACKBONE.LAYERS_LR * cfg.TRAIN.BASE_LR}]

    trainable_params += [{'params': model.rpn_head.parameters(),
                          'lr': cfg.TRAIN.BASE_LR}]

optimizer = torch.optim.SGD(trainable_params,
                                momentum=cfg.TRAIN.MOMENTUM,
                                weight_decay=cfg.TRAIN.WEIGHT_DECAY)

In aboved these codes, after I analyse the lr_scheduler codes, it seems that no use to define lr in trainable_params, because the trainable params' lr will be covered by lr_scheduler.step(). look forward to your reply!

[ZhiyuanChen]

They all point to the same object.