exp push: fails

[dberenbaum]

Bug Report

Description

dvc exp push is failing.

Reproduce

1. Fork https://github.com/iterative/example-dvc-experiments and clone the fork.
2. Run experiments.
3. Try to push experiments.

Output:

$ dvc exp push origin exp-b270f
2021-11-02 20:08:40,190 DEBUG: git push experiment 'refs/exps/06/eed83b30c1e3d6cd7be76c0965d7e0ea56439e/exp-b270f' -> 'origin'
2021-11-02 20:08:40,233 ERROR: unexpected error - [Errno 49] Can't assign requested address
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/Users/dave/Code/dvc/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/Users/dave/Code/dvc/dvc/command/experiments.py", line 728, in run
    self.repo.experiments.push(
  File "/Users/dave/Code/dvc/dvc/repo/experiments/__init__.py", line 1003, in push
    return push(self.repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/scm_context.py", line 14, in run
    return method(repo, *args, **kw)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/push.py", line 40, in push
    repo.scm.push_refspec(
  File "/Users/dave/Code/dvc/dvc/scm/git/__init__.py", line 296, in _backend_func
    return func(*args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/scm/git/backend/dulwich/__init__.py", line 452, in push_refspec
    client.send_pack(
  File "/opt/homebrew/lib/python3.9/site-packages/dulwich/client.py", line 926, in send_pack
    proto, unused_can_read, stderr = self._connect(b"receive-pack", path)
  File "/opt/homebrew/lib/python3.9/site-packages/dulwich/client.py", line 1670, in _connect
    con = self.ssh_vendor.run_command(
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/Users/dave/Code/dvc/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 80, in _run_command
    conn = await asyncssh.connect(
  File "/opt/homebrew/lib/python3.9/site-packages/asyncssh/connection.py", line 6855, in connect
    return await _connect(options, loop, flags, conn_factory,
  File "/opt/homebrew/lib/python3.9/site-packages/asyncssh/connection.py", line 297, in _connect
    _, conn = await loop.create_connection(conn_factory, host, port,
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1056, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1041, in create_connection
    sock = await self._connect_sock(
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 955, in _connect_sock
    await self.sock_connect(sock, address)
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/selector_events.py", line 502, in sock_connect
    return await fut
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/selector_events.py", line 507, in _sock_connect
    sock.connect(address)
OSError: [Errno 49] Can't assign requested address
------------------------------------------------------------
2021-11-02 20:08:40,333 DEBUG: Version info for developers:
DVC version: 2.8.3.dev13+g5690015e
---------------------------------
Platform: Python 3.9.5 on macOS-12.0.1-arm64-arm-64bit
Supports:
        webhdfs (fsspec = 2021.9.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        s3 (s3fs = 2021.8.0, boto3 = 1.17.5)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-11-02 20:08:40,333 DEBUG: Analytics is disabled.

[karajan1001]

I tried a self push and it succeeded.

dvc exp push . exp-96581
Pushed experiment 'exp-96581'to Git remote '.'.

But can not push to the origin, maybe because of permission? and the strange thing is that my error is different from yours.

2021-11-04 15:15:00,563 ERROR: unexpected error - Connection lost
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gao/Code/dvc/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/Users/gao/Code/dvc/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/Users/gao/Code/dvc/dvc/command/experiments.py", line 728, in run
    self.repo.experiments.push(
  File "/Users/gao/Code/dvc/dvc/repo/experiments/__init__.py", line 1003, in push
    return push(self.repo, *args, **kwargs)
  File "/Users/gao/Code/dvc/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/gao/Code/dvc/dvc/repo/scm_context.py", line 14, in run
    return method(repo, *args, **kw)
  File "/Users/gao/Code/dvc/dvc/repo/experiments/push.py", line 40, in push
    repo.scm.push_refspec(
  File "/Users/gao/Code/dvc/dvc/scm/git/__init__.py", line 296, in _backend_func
    return func(*args, **kwargs)
  File "/Users/gao/Code/dvc/dvc/scm/git/backend/dulwich/__init__.py", line 452, in push_refspec
    client.send_pack(
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/dulwich/client.py", line 917, in send_pack
    proto, unused_can_read, stderr = self._connect(b"receive-pack", path)
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/dulwich/client.py", line 1661, in _connect
    con = self.ssh_vendor.run_command(
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/Users/gao/Code/dvc/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 80, in _run_command
    conn = await asyncssh.connect(
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/asyncssh/connection.py", line 6803, in connect
    return await _connect(options, loop, flags, conn_factory,
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/asyncssh/connection.py", line 303, in _connect
    await conn.wait_established()
  File "/Users/gao/anaconda3/envs/dvc/lib/python3.8/site-packages/asyncssh/connection.py", line 2243, in wait_established
    await self._waiter
asyncssh.misc.ConnectionLost: Connection lost

[dberenbaum]

Hmm, on the one hand, that makes it pretty likely this is a personal issue and not a DVC one, but on the other I wonder why it's so flaky?

Can you do dvc exp push --no-cache origin exp-96581? This still fails for me.

I can push via http like dvc exp push --no-cache https://dberenbaum:<token>@github.com/dberenbaum/dvc-checkpoints-mnist.git exp-b6560.

[karajan1001]

Hmm, on the one hand, that makes it pretty likely this is a personal issue and not a DVC one, but on the other I wonder why it's so flaky?

Can you do dvc exp push --no-cache origin exp-96581? This still fails for me.

I can push via http like dvc exp push --no-cache https://dberenbaum:<token>@github.com/dberenbaum/dvc-checkpoints-mnist.git exp-b6560.

Behave the same, because the cache in --no-cache here is about the dvc cache, and the failure comes from the git permission.

[dberenbaum]

Right, just checking. So you are consistently getting the connection lost error? And is it the same for you with ssh or http?

[karajan1001]

The exactly same error for me.

[iesahin]

I get this error in dvc exp list and dvc exp push when I use ssh URLs for a Git repository. When I use https URLs, dvc exp list works fine.

The error persists on 2.8.3. AFAIR these work fine in 2.7.

[iesahin]

Any updates on this?

[efiop]

Likely caused by #6797

[dberenbaum]

Confirmed that exp push works for me over ssh prior to #6797

[dberenbaum]

This breaks pushing experiments over ssh, which is our only supported way to authenticate git remotes (https://github.com/iterative/dvc.org/pull/2908/files), and we have a release planned for next week to promote experiments, so I'm setting this to p0.

[efiop]

For the record: same issue with just running dvc exp list origin (just much faster to reproduce than actually running an experiment)

EDIT: paramiko vendor seems to work fine, something is up with our async one. Need to leave till tomorrow evening, will take a look then.