Optimizers for coord check

[xwjabc]

Thank you for your great work! When trying the coord check in the examples, I noticed that the original optimizers (e.g., sgd, adam) are used instead of the muP optimizers (e.g., musgd, muadam). However, according to the Table 8 in the paper, the optimizers should be adjusted accordingly to make activations bounded. Is there any reason behind the use of original optimizers?

[edwardjhu]

Hi Weijian,

The mup coordinate check curves do use mu-optimizers. The conversion happens internally:

mup/mup/coord_check.py

Line 441 in eac6f1d

if mup:

Does this address your concern?

[xwjabc]

Thank you for your quick reply! Yes, it addresses my concern. I just found out the auto conversion right before seeing your reply 😆