About the optimizer and scheduler

[Pay20Y]

There seems to be no code call the function "CustomCosineAnnealingWarmupRestarts"? I wonder if the optimizer and scheduler you used are AdamW and Cosine Annealing?
Thanks!

[YongWookHa]

The CustomCosineAnnealingWarmupRestarts Code is in utils.py.

swin-transformer-ocr/utils.py

Line 44 in 1970e8d

class CustomCosineAnnealingWarmupRestarts(_LRScheduler):

Have a look at it, plz :)

[Pay20Y]

Thanks for you quick reply. My question is that there seems to be no code use the function "CustomCosineAnnealingWarmupRestarts", are there some special usages in the pytorch_lightning?

[YongWookHa]

Oh, I guess I misunderstood your question.

swin-transformer-ocr/models.py

Line 57 in 1970e8d

scheduler = getattr(utils, self.cfg.scheduler)

Here you can find the usage of CustomCosineAnnealingWarmupRestarts .

It depends on what you wrote in your yaml file which contains the settings.

swin-transformer-ocr/settings/default.yaml

Line 14 in 1970e8d

scheduler: "CustomCosineAnnealingWarmupRestarts"

As you can check, default.yaml says it will use CustomCosineAnnealingWarmupRestarts .

This image shows how the learning rate moves with CustomCosineAnnealingWarmupRestarts .

I wrote a tech post in my blog. Though it's in Korean language, you can use browser translation with chrome or edge:)

Hope this helps you.

[Pay20Y]

I got it, thanks again!