About Learning Rate Scheduler

[1049451037]

❔Question

Why the step of learning rate scheduler after each epoch instead of each batch in main.py?

Won't the change rate of lr be too slow? (and unstable for various dataset sizes)

[Yuxin-CV]

Hi, Qingsong. Thanks for this issue.

To my knowledge, in image recognition, the lr is usually stepped by epoch if you choose the cosine lr scheduler, e.g., in the widely used timm library.

It seems that in NLP the lr scheduler is stepped after each iteration / batch. e.g., in the BEiT repo. This is also true for semantic segmentation in vision.

I agree with you that step by iteration is more reasonable than step by epoch, and step by iteration should yield no worse results than step by epoch.

[1049451037]

Got it. Thank you for your reply!