[optim][foreach] Do NOT inplace modify gradients

[janeyx99]

SGD and ASGD already had out-of-place grads.

Stack from ghstack (oldest at bottom):

• [optim][nadam] group tensors in foreach, make it default #92715
• -> [optim][foreach] Do NOT inplace modify gradients #92706
• [optim][NAdam] Fix discrepancy between mt vs st impl #92699

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92706

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a592359:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[janeyx99]

cc'ing @ngimel to verify the perf perspective

[janeyx99]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here