Introduce the torchrun entrypoint

[cbalioglu]

This PR introduces a new torchrun entrypoint that simply "points" to python -m torch.distributed.run. It is shorter and less error-prone to type and gives a nicer syntax than a rather cryptic python -m ... command line. Along with the new entrypoint the documentation is also updated and places where torch.distributed.run are mentioned are replaced with torchrun.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23

[facebook-github-bot]

🔗 Helpful links

• 🧪 See artifacts and rendered test results at hud.pytorch.org/pr/64049
• 📄 Preview docs built from this PR

💊 CI failures summary and remediations

As of commit 6e17981 (more details on the Dr. CI page):

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[facebook-github-bot]

@kiukchung has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@kiukchung merged this pull request in 65e6194.

[vadimkantorov]

One controversial thing about it is that it hides somewhat which Python distribution is used for running.

It may be good that it prints the path to used Python on the screen along with an explanation of how to change it if needed. E.g. for the case where python is aliased to system's python2 and the good one is python3.

It may also be good for the script to have some guards against old or unrelated pythons

Also, could not find source of torchrun in PyTorch sources. Is it exactly equivalent to python -m torch.distributed.run? If so, it should be clearly stated in the docs. Currently, it may leave an impression is that torchrun has some extended functionality / magic (esp. phrasing torchrun is superset of python -m torch.distributed.run). Which along with difficulty of finding the source code raises questions