Document change for DDP enabled on Windows platform #45392

gunandrose4u · 2020-09-27T08:23:47Z

Document change for DDP enabled on Windows platform

Update my fork from Pytorch repo

Rebase from pytorch/pytorch master

Rebase latest code from master branch of PyTorch repo

Rebase code

Rebase to latest commits

Rebase to latest commit

Rebase to latest commits

Rebase latest commits

dr-ci · 2020-09-27T09:06:51Z

💊 CI failures summary and remediations

As of commit 30ad222 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-CircleCI failure(s)

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.7-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 7 times.

codecov · 2020-09-27T12:49:08Z

Codecov Report

Merging #45392 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #45392   +/-   ##
=======================================
  Coverage   68.15%   68.15%           
=======================================
  Files         396      396           
  Lines       51133    51133           
=======================================
+ Hits        34849    34850    +1     
+ Misses      16284    16283    -1

Impacted Files	Coverage Δ
torch/distributed/__init__.py	`87.50% <ø> (ø)`
torch/testing/_internal/expecttest.py	`78.57% <0.00%> (+1.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc5710f...30ad222. Read the comment docs.

mrshenli · 2020-09-27T16:49:17Z

README.md

+```bash
+# Add these packages and set libuv_ROOT environment variable if torch.distributed is needed
+conda install -y -q -c rdonnelly libuv
+set libuv_ROOT={conda active env location}\Library


I wonder why MacOS does not require this? MacOS requires install pkg-config instead. Can we do the same for Windows?

In Gloo PR #45392, I first tried with pkg-config, but Windows doesn't come with pkg-config, later on, base on the code review suggestion, I switched to use find_library and find_file to work with libuv_ROOT. So libuv_ROOT need to be set as a environment variable, or defined in cmake command line as -Dlibuv_ROOT.

but Windows doesn't come with pkg-config, later on, base on the code review suggestion

Does this mean even if we do conda install pkg-config libuv, we still cannot find libuv on Windows automatically?

mrshenli · 2020-09-28T14:09:51Z

README.md


+On Windows
+```bash
+# Add these packages and set libuv_ROOT environment variable if torch.distributed is needed


Let's add one line:

# Distributed package support on Windows is a prototype feature and is subject to changes.

@jspisak @gchanan is it OK to mention a prototype feature in master README.md?

mrshenli · 2020-09-28T14:11:36Z

docs/source/distributed.rst

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-PyTorch distributed currently only supports Linux. By default, the Gloo and NCCL backends
+PyTorch distributed currently only supports Linux and Windows. By default for Linux, the Gloo and NCCL backends


"PyTorch distributed currently only supports Linux and Windows." -> "PyTorch distributed package supports Linux (stable), MacOS (stable), and Windows (prototype)."

mrshenli · 2020-09-28T14:15:51Z

docs/source/distributed.rst

 optional backend that can only be included if you build PyTorch from source. (e.g.
 building PyTorch on a host that has MPI installed.)

+.. note ::


Let's use .. warning:: instead of .. note::

mrshenli · 2020-09-28T14:26:04Z

docs/source/distributed.rst

 building PyTorch on a host that has MPI installed.)

+.. note ::
+    On Windows, Pytorch distributed currently only supports Gloo backend, and FileStore as


As of PyTorch v1.7, Windows support for the distributed package only covers collective communications with Gloo backend, `FileStore`, and `DistributedDataParallel`. Therefore, the `init_method` argument in :func:`init_process_group` must point to a file. This works for both local and shared file systems: local file system example ``init_method="file:///d:/tmp/some_file"``, shared file system example ``init_method="file://////{machine_name}/{share_folder_name}/some_file"``. Similarly, if you directly pass in a `store` argument, it must be a ``FileStore`` instance.

mrshenli · 2020-09-28T14:27:22Z

torch/distributed/__init__.py

+    ``torch.distributed`` is available on Linux, MacOS and Windows. Set
    ``USE_DISTRIBUTED=1`` to enable it when building PyTorch from source.
-    Currently, the default value is ``USE_DISTRIBUTED=1`` for Linux and
+    Currently, the default value is ``USE_DISTRIBUTED=1`` for Linux and Windows,


I wonder if we should make USE_DISTRIBUTED=0 by default, and let users set it to 0 when necessary?

facebook-github-bot

@mrshenli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@mrshenli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-09-28T22:30:41Z

@mrshenli merged this pull request in 47debdc.

gunandrose4u and others added 13 commits August 17, 2020 11:27

Merge pull request #1 from pytorch/master

bf02d77

Update my fork from Pytorch repo

Merge pull request #2 from pytorch/master

2ebe620

Rebase from pytorch/pytorch master

Merge pull request #3 from pytorch/master

04f7b51

Rebase latest code from master branch of PyTorch repo

Merge pull request #4 from pytorch/master

afbc6c1

Rebase code

Merge pull request #5 from pytorch/master

c933f32

Rebase to latest commits

Merge pull request #6 from pytorch/master

ddd80bc

Rebase to latest commits

Merge pull request #7 from pytorch/master

cd3d2af

Rebase to latest commit

Merge pull request #8 from pytorch/master

450742a

Rebase to latest commits

Merge pull request #9 from pytorch/master

34ed95a

Rebase to latest commits

Merge pull request #10 from pytorch/master

a6d5771

Rebase to latest commits

Merge pull request #13 from pytorch/master

96927c0

Rebase latest commits

Merge pull request #14 from pytorch/master

9147d60

Rebase latest commits

Update document for enable DDP on Windows platform

4d06d2c

gunandrose4u requested review from apaszke, mrshenli, pietern, pritamdamania87, rohan-varma and zhaojuanmao as code owners September 27, 2020 08:23

pytorchbot added the open source label Sep 27, 2020

mrshenli reviewed Sep 27, 2020

View reviewed changes

mrshenli reviewed Sep 28, 2020

View reviewed changes

facebook-github-bot reviewed Sep 28, 2020

View reviewed changes

gunandrose4u added 2 commits September 28, 2020 22:59

Resolve code review comments

f74746a

Format list

30ad222

facebook-github-bot reviewed Sep 28, 2020

View reviewed changes

mrshenli approved these changes Sep 28, 2020

View reviewed changes

facebook-github-bot closed this in 47debdc Sep 28, 2020

facebook-github-bot added the merged label Sep 28, 2020

mruberry added the Merged label Oct 28, 2020

Document change for DDP enabled on Windows platform #45392

Document change for DDP enabled on Windows platform #45392

Uh oh!

Conversation

gunandrose4u commented Sep 27, 2020

Uh oh!

dr-ci bot commented Sep 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

ci.pytorch.org: 1 failed

Uh oh!

codecov bot commented Sep 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dr-ci bot commented Sep 27, 2020 •

edited

Loading

codecov bot commented Sep 27, 2020 •

edited

Loading