Include lr scaling info in adamw weight_decay docstring

[thomas-woehrle]

Adds documentation as discussed in #38853

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154113

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 21a5219 with merge base d7a83ab ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[janeyx99]

I have two concerns, but maybe it's okay.

1. While I like having some mention of this over none at all, I'm not sure the doc reader will necessarily know what this means from just the phrase. It may be better to include a bigger note:: and include the mathematical difference like mentioned in the issue.
2. We should mention this in all the *AdamWs, NAdam + RAdam as they have decoupled_weight_decay options.

[thomas-woehrle]

1. While I like having some mention of this over none at all, I'm not sure the doc reader will necessarily know what this means from just the phrase. It may be better to include a bigger note:: and include the mathematical difference like mentioned in the issue.

One could link to the comment of the original implementation issue, here. It explains the "problem" nicely. Would this be preferable over a description in the docstring?

2. We should mention this in all the *AdamWs, NAdam + RAdam as they have decoupled_weight_decay options.

I could add the note to all of them. This would make it even more desirable to link to an existing explanation compared to explaining the issue in detail in the docstring itself.

Also, if #22343 would be implemented things would/could change again, just mentioning that

[janeyx99]

#22343 will likely not be implemented any time soon.

A description in the docstr is still preferred (as a digestible description) and I'm not sure we typically link to issues in docs (though I'm not theoretically opposed). You can define a docstr in optimizer.py and import it in all the relevant files for deduplication.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.