FR: Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term (WSAM)

[LiutongZhou]

Paper: https://arxiv.org/pdf/2305.15817.pdf
Code: https://github.com/intelligent-machine-learning/dlrover/blob/master/atorch/atorch/optimizers/wsam.py

Benefits over SAM: lower loss, higher accuracy, and better generalization (lower sharpness) at the same time.

Insensitive to the additional hyperparameter $\rho$. Optimal value between 0.8 ~ 0.95

[kozistr]

thanks! I'll check it too.

[kozistr]

I just merged into the main branch. (not released a new version yet)

thanks for the suggestion!