Clarification regarding adding 1e-12 while calculating the scale

[sayakpaul]

Hi @davda54. Thank you for open-sourcing this implementation.

Wanted to know the reason behind adding 1e-12 in https://github.com/davda54/sam/blob/main/sam.py#L18?

[davda54]

Hi, thanks! :) A small positive number is added to the denominator for numerical stability — to avoid division by zero when grad_norm == 0.0.

[sayakpaul]

I see. That is what I had thought too. Thank you for confirming.

I am also assuming your e_w calculation is with respect to L2 norm as the authors assert that they get the optimal results with that?

[davda54]

Exactly, I assume that p == q == 2 (similarly to the paper), which simplifies the equations.