EtcdStore: No constructor defined

[kit1980]

Hi,

I've updated my torchelastic to latest (including 393a26c commit) and PyTorch to 1.4.

My test setup used to work OK with TCPStore, now I get an error:

INFO 2020-01-23 01:39:31,128 Creating EtcdStore as the c10d::Store implementation
[ERROR] 2020-01-23 01:39:31,139 coordinator_p2p: Rank: -1
Error: Rank -1 received an Exception. Detailed message: EtcdStore: No constructor defined!
ErrorType: <class 'torchelastic.coordinator.NonRetryableException'>
StackTrace: Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.6/site-packages/torchelastic-0.1.0rc2-py3.6.egg/torchelastic/p2p/coordinator_p2p.py", line 64, in rendezvous_barrier
    self.store, self.rank, self.world_size = self.rendezvous.next_rendezvous()
  File "/opt/miniconda/lib/python3.6/site-packages/torchelastic-0.1.0rc2-py3.6.egg/torchelastic/rendezvous/etcd_rendezvous.py", line 98, in next_rendezvous
    store = self._rdzv_impl.setup_kv_store(rdzv_version)
  File "/opt/miniconda/lib/python3.6/site-packages/torchelastic-0.1.0rc2-py3.6.egg/torchelastic/rendezvous/etcd_rendezvous.py", line 851, in setup_kv_store
    return EtcdStore(etcd_client=self.client, etcd_store_prefix=store_path)
  File "/opt/miniconda/lib/python3.6/site-packages/torchelastic-0.1.0rc2-py3.6.egg/torchelastic/rendezvous/etcd_rendezvous.py", line 865, in __init__
    super().__init__()  # required for pybind trampoline.
TypeError: EtcdStore: No constructor defined!

Is EtcdStore should to be ready to use now or some code updates are still needed?

[kiukchung]

Hey, yea I realized this as well. Turns out the commit we need was not included in pytorch v1.4.0 release.... This is the commit we need
pytorch/pytorch@1d3f3a1

Running
git tag --contains 1d3f3a1a0cc0f66b51d8e8f4006fb200e18e2818
on the pytorch git repo doesn't return anything....

I'm shuffling through the torch nightly release to see if taking a temporary dependency on the nightly until the next torch release is an option. Otherwise I would have to revert this back and help you figure out the dns issue or change the gethostname to getfqdn

[kiukchung]

I validated that this works if we take a dependency on the torch nightly build.

pip install --pre torch -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html

I think this unblocks you immediately. I'll have a fix out for setup.py so that we can just take a dependency on torch-nightly for the time being until torch-1.5 is released.

[kit1980]

Thanks, I'll try with that PyTorch commit.
I'm personally totally OK with nightly PyTorch build requirement for now, but maybe you should mention this prominently for others potentially trying torchelastic.

[kiukchung]

FYI, here's the exact version of torch I validated with:

ubuntu@ip-172-31-26-180:~/workspace/elastic$ pip show torch
Name: torch
Version: 1.5.0.dev20200122
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /home/ubuntu/anaconda3/lib/python3.6/site-packages
Requires:
Required-by: torchelastic

[kit1980]

Thank you, I've verified on my setup that it works with PyTorch nightly.