torch.distributed support on MacOS is missing

[yaroslavvb]

Currently trying to use distributed on MacOS crashes because torch.distributed namespace is empty
I vaguely recall it working a year ago.

This is useful for quick sanity checks on my MacBook before deploying to cluster.

Python 3.7.3 (default, Mar 27 2019, 16:54:48) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.version.__version__)
1.1.0
>>> import torch.distributed as dist
>>> dist.init_process_group
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.distributed' has no attribute 'init_process_group'

[jeffreyksmithjr]

Cc @pietern @mrshenli

[soumith]

@yaroslavvb from what I remember, we dont plan to add distributed support on OSX or Windows

[yaroslavvb]

How hard would it be to add 1-worker DDP support to Mac? For instance -m torch.distributed.launch already works.

This would make it easier for people who prototype on Macs before deploying to Linux. Currently it's a bit annoying with if sys.platform=="darwin" bits cropping up everywhere, example

cc @jspisak who would know if there's enough people using PyTorch on Mac. From my Bay Area vantage, it seems to be everybody.

[pietern]

@yaroslavvb Good point. I agree. This has been raised before and I agree we should revisit.

Adding macOS support (and Windows support for that matter) will be contingent on making the Gloo TCP transport work with something other than epoll(2), which is the Linux specific part. This can be done by either adding kqueue support to gloo/transport/tcp/device.cc, or by adding another transport that doesn't have any platform specifics at all (e.g. through libuv).

[Ownmarc]

+1 for Windows support of the distributed training! Currently getting "AttributeError: module 'torch.distributed' has no attribute 'init_process_group'" when trying to run distributed training on Windows. The excat same code works fine on Linux.

[pietern]

@Ownmarc In what kind of environment would you use this? Multiple machine or only multiple processes on a single machine? I have not heard of folks wanting to use torch.distributed on Windows so am curious to hear more about your intended usage.

[Ownmarc]

@Ownmarc In what kind of environment would you use this? Multiple machine or only multiple processes on a single machine? I have not heard of folks wanting to use torch.distributed on Windows so am curious to hear more about your intended usage.

To take advantage of 2 GPUs in the same machine.

@pietern here is an other person. I am using this repo too ultralytics/yolov3#336 (comment)