AdaBound

[r1ckya]

Hi, thanks for great work.
Would you also consider checking out AdaBound optimizer? It claims to be

as fast as Adam and as good as SGD

It seems worth to put it to the test and combine with other techniques.
https://github.com/Luolc/AdaBound

[mgrankin]

Hello, have you used AdaBound? Is there any rule of thumb on how to tune final_lr?

[r1ckya]

I didn't yet. Not that I'm aware of, but final_lr is in a sense SGD lr and lr is Adam lr. So maybe running lr_finder with SGD first makes sense.

[mgrankin]

Have you tried to run AdaBound? I'm getting

ZeroDivisionError: float division by zero

It seems that base_lrs aren't initialised properly. What should they be?

[r1ckya]

Yes, I have. Didn't see this error. I have used this version.
https://github.com/Luolc/AdaBound/blob/master/adabound/adabound.py
base_lrs shouldn't be zero because group['lr'] != 0
self.base_lrs = list(map(lambda group: group['lr'], self.param_groups))

[mgrankin]

I think the error I've received was due to LR scheduling. @r1ckya which LR schedule do you use?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.