TCPStore: Address already in use in test_distributed

[pietern]

🐛 Bug

Flaky execution of test_distributed due to port conflicts. This is an intermittent issue.

Can happen either due to the same process interference (the subprocesses not exiting fast enough before the next test is started -- need to double check if this is possible at all), or due to multi process interference, e.g. when the same tests run on the same machine at the same time and tries to bind to the same port.

We can fix by making these tests use the FileStore on a unique path instead.

cc @ezyang @teng-li @ssnl

[pietern]

See e.g. https://circleci.com/gh/pytorch/pytorch/87644

The first pass to fixing this was merged in #12782 but only addressed test_c10d.py.

[pietern]

Output of that particular test:

Oct 19 12:27:22 test_reduce_multigpu (__main__.TestDistBackend) ... Process process 0:
Oct 19 12:27:22 Traceback (most recent call last):
Oct 19 12:27:22   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Oct 19 12:27:22     self.run()
Oct 19 12:27:22   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
Oct 19 12:27:22     self._target(*self._args, **self._kwargs)
Oct 19 12:27:22   File "test_distributed.py", line 1330, in _run
Oct 19 12:27:22     rank=self.rank
Oct 19 12:27:22   File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 280, in init_process_group
Oct 19 12:27:22     store, rank, world_size = next(rendezvous(init_method))
Oct 19 12:27:22   File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 131, in _env_rendezvous_handler
Oct 19 12:27:22     store = TCPStore(master_addr, master_port, start_daemon)
Oct 19 12:27:22 RuntimeError: Address already in use
Oct 19 12:37:22 FAIL

The failure can happen on any test that uses the TCP store though, so removing all uses will fix the problem.