Schedule Free Optimizer

[t0278611]

Schedule Free Optimizer (SFO) has proved quite strong for different tasks (e.g. in kaggle competitions), giving strong out of the box performance and fast convergence.

Has anyone tried using SFO for image models? Especially for baselines it seems like a good way save some tuning

[vincere-mori]

Tried it on a few classification tasks. It works, but the thing that trips people up most: you have to call optimizer.eval() before validation. SFO maintains two parameter sets internally (the base and the EMA-like scheduled weights), and model.eval() alone does not switch them. Missing this gives you garbage val metrics.

Performance-wise it matched well-tuned cosine decay + warmup on ResNets without any LR search, which is the whole point. Makes sense as a default for baselines and quick experiments.

One caveat with the timm training script: a lot of the scheduler logic (--sched cosine, --warmup-epochs etc.) becomes irrelevant since SFO replaces the schedule entirely. The SFO paper recommends a short linear warmup still, but you configure that inside the optimizer itself, not through timm's scheduler. Easiest to just disable --sched and let SFO handle it.