[Feature] Implement REX lr scheduler

[kozistr]

Problem (Why?)

related to #217

Solution (What/How?)

• implement REX lr scheduler

Other changes (bug fixes, small refactors)

N/A

Notes

N/A

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (fd717fc) to head (a3b1bfb).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #222   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           81        82    +1     
  Lines         5912      5946   +34     
=========================================
+ Hits          5912      5946   +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[clementpoiret]

Hi!
For the sake of curiosity, why does the impl differs from the original paper?
Shouldn't it be self.max_lr * ((1.0 - progress) / (.5 + (1 - progress) / 2.0))?

[kozistr]

Hi! For the sake of curiosity, why does the impl differs from the original paper? Shouldn't it be self.max_lr * ((1.0 - progress) / (.5 + (1 - progress) / 2.0))?

sorry for the late response, i missed it

as you mentioned, the formula self.max_lr * ((1.0 - progress) / (.5 + (1 - progress) / 2.0)) is right! actually, it is equal to my implementation (min_lr + (max_lr - min_lr) * ((1.0 - progress) / (1.0 - progress / 2.0))) when min_lr is 0.

the reason why i added min_lr is for the flexibility and further usage of the REX scheduler : )

hope it helps you!

[clementpoiret]

Ah great! I understand, thanks for your reply :)