dvc exp list origin: unexpected error - [Errno 111] Connect call failed

[cbracegirdle]

Bug Report

Description

I have a simple setup with a git repo and single HDFS dvc remote. DVC is correctly pushing to the remote. However when I try to use dvc exp to list experiments from the git repo, I see a connection error.

Reproduce

dvc exp list origin

Expected

List of experiments as shown at https://dvc.org/doc/user-guide/experiment-management/sharing-experiments

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 2.8.3 (pip)
---------------------------------
Platform: Python 3.7.10 on Linux-4.18.0-15-generic-x86_64-with-debian-buster-sid
Supports:
	hdfs (fsspec = 2021.11.0, pyarrow = 3.0.0),
	webhdfs (fsspec = 2021.11.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: zfs on zpmain/home
Caches: local
Remotes: hdfs
Workspace directory: zfs on zpmain/home
Repo: dvc, git

Additional Information (if any):

$ dvc exp list origin -v
2021-11-25 11:27:03,065 ERROR: unexpected error - [Errno 111] Connect call failed (<COMPANY_IP>, 0)
------------------------------------------------------------
Traceback (most recent call last):
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/command/experiments.py", line 707, in run
    all_=self.args.all,
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/__init__.py", line 1013, in ls
    return ls(self.repo, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/scm_context.py", line 14, in run
    return method(repo, *args, **kw)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/ls.py", line 40, in ls
    for info in gen:
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/utils.py", line 76, in remote_exp_refs_by_baseline
    for ref in scm.iter_remote_refs(url, base=str(ref_info)):
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/scm/git/backend/dulwich/__init__.py", line 379, in iter_remote_refs
    for ref in client.get_refs(path)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dulwich/client.py", line 1068, in get_refs
    proto, _, stderr = self._connect(b"upload-pack", path)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dulwich/client.py", line 1688, in _connect
    self.host, argv, port=self.port, username=self.username, **kwargs
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 87, in _run_command
    encoding=None,
  File "/home2/spark/spark_env/lib/python3.7/site-packages/asyncssh/connection.py", line 6895, in connect
    timeout=options.connect_timeout)
  File "/opt/conda/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
    return await fut
  File "/home2/spark/spark_env/lib/python3.7/site-packages/asyncssh/connection.py", line 300, in _connect
    local_addr=local_addr)
  File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 962, in create_connection
    raise exceptions[0]
  File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 949, in create_connection
    await self.sock_connect(sock, address)
  File "/opt/conda/lib/python3.7/asyncio/selector_events.py", line 473, in sock_connect
    return await fut
  File "/opt/conda/lib/python3.7/asyncio/selector_events.py", line 503, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed (<COMPANY_IP>, 0)
------------------------------------------------------------

[efiop]

@cbracegirdle Hi. Could you try pip install -U "git+https://github.com/iterative/dvc#egg=dvc[ssh]" and give it a try? Also, are you able to use CLI git to, for example, clone the origin?

[cbracegirdle]

Thanks, that helped and I now see:

2021-11-25 13:33:11,465 ERROR: unexpected error - Permission denied   
------------------------------------------------------------
Traceback (most recent call last):
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/command/experiments.py", line 710, in run
    all_=self.args.all,
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/__init__.py", line 1018, in ls
    return ls(self.repo, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/scm_context.py", line 149, in run
    return method(repo, *args, **kw)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/ls.py", line 40, in ls
    for info in gen:
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/utils.py", line 111, in remote_exp_refs_by_baseline
    for ref in iter_remote_refs(scm, url, base=str(ref_info)):
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/repo/experiments/utils.py", line 59, in iter_remote_refs
    yield from scm.iter_remote_refs(url, base=base, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/scm/git/backend/dulwich/__init__.py", line 381, in iter_remote_refs
    for ref in client.get_refs(path)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dulwich/client.py", line 1068, in get_refs
    proto, _, stderr = self._connect(b"upload-pack", path)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dulwich/client.py", line 1688, in _connect
    self.host, argv, port=self.port, username=self.username, **kwargs
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home2/spark/spark_env/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home2/spark/spark_env/lib/python3.7/site-packages/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 139, in _run_command
    encoding=None,
  File "/home2/spark/spark_env/lib/python3.7/site-packages/asyncssh/connection.py", line 6895, in connect
    timeout=options.connect_timeout)
  File "/opt/conda/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
    return await fut
  File "/home2/spark/spark_env/lib/python3.7/site-packages/asyncssh/connection.py", line 303, in _connect
    await conn.wait_established()
  File "/home2/spark/spark_env/lib/python3.7/site-packages/asyncssh/connection.py", line 2262, in wait_established
    await self._waiter
asyncssh.misc.PermissionDenied: Permission denied

I am using an ssh key with passphrase for git access. I am able to push/pull/clone with git.

[cbracegirdle]

I fixed it by using ssh-agent. Thanks for your help!

[efiop]

@cbracegirdle Could you elaborate, please, on how you've fixed it? Trying to see if there is anything we can do here on dvc side to improve.

[cbracegirdle]

Sure, based on https://askubuntu.com/a/362287

I ran

$ ssh-agent bash

then

$ ssh-add /home/username/.ssh/id_rsa
Enter passphrase for /home/username/.ssh/id_rsa: 
Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa)
$ dvc exp list origin