dulwich: fallback to default SSH keys in asyncssh vendor #6983

pmrowla · 2021-11-15T11:03:16Z

❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Will fix #6920

Related to #2215

pmrowla · 2021-11-15T11:12:21Z

@dberenbaum @karajan1001 can you try this patch to confirm it allows you to push/list exps from github? Also please note what type of SSH keys you are using (RSA or ed25519)

pmrowla · 2021-11-15T11:22:07Z

dvc/scm/git/backend/dulwich/asyncssh_vendor.py

Should note that this appears to be specific to github. gitlab's SSH server returns the proper algorithm names in the auth responses.

This is also one of those cases where the RFC's do note that some existing SSH implementations in the wild may just always use "ssh-rsa", and that it's up to individual client/server implementations to decide whether or not to allow or reject those requests & responses.

karajan1001 · 2021-11-15T13:13:28Z

My result

$ dvc exp push origin name -vv
2021-11-15 21:12:48,539 TRACE: Namespace(cd='.', cmd='push', cprofile=False, cprofile_dump=None, dvc_remote=None, experiment='name', force=False, func=<class 'dvc.command.experiments.CmdExperimentsPush'>, git_remote='origin', instrument=False, instrument_open=False, jobs=None, pdb=False, push_cache=True, quiet=0, run_cache=False, verbose=2, version=None, yappi=False)
2021-11-15 21:12:48,624 DEBUG: git push experiment 'refs/exps/57/e8d6f48431025a93410b21ba4c8943115fcc26/name' -> 'origin'
2021-11-15 21:12:52,417 ERROR: unexpected error - 'NoneType' object has no attribute 'pause_writing'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gyx/code/github/dvc/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/gyx/code/github/dvc/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/gyx/code/github/dvc/dvc/command/experiments.py", line 731, in run
    self.repo.experiments.push(
  File "/home/gyx/code/github/dvc/dvc/repo/experiments/__init__.py", line 1001, in push
    return push(self.repo, *args, **kwargs)
  File "/home/gyx/code/github/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gyx/code/github/dvc/dvc/repo/scm_context.py", line 14, in run
    return method(repo, *args, **kw)
  File "/home/gyx/code/github/dvc/dvc/repo/experiments/push.py", line 40, in push
    repo.scm.push_refspec(
  File "/home/gyx/code/github/dvc/dvc/scm/git/__init__.py", line 296, in _backend_func
    return func(*args, **kwargs)
  File "/home/gyx/code/github/dvc/dvc/scm/git/backend/dulwich/__init__.py", line 452, in push_refspec
    client.send_pack(
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/dulwich/client.py", line 977, in send_pack
    write_pack_data(proto.write_file(), pack_data_count, pack_data)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/dulwich/pack.py", line 1744, in write_pack_data
    crc32 = write_pack_object(f, type_num, raw, compression_level=compression_level)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/dulwich/pack.py", line 1583, in write_pack_object
    f.write(data)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/dulwich/pack.py", line 1512, in write
    self.f.write(data)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/dulwich/protocol.py", line 316, in write
    self._proto.write(data)
  File "/home/gyx/code/github/dvc/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 45, in write
    self.proc.stdin.write(data)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/asyncssh/stream.py", line 293, in write
    return self._chan.write(data, self._datatype)
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/asyncssh/channel.py", line 861, in write
    self._flush_send_buf()
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/asyncssh/channel.py", line 280, in _flush_send_buf
    self._pause_resume_writing()
  File "/home/gyx/anaconda3/lib/python3.8/site-packages/asyncssh/channel.py", line 255, in _pause_resume_writing
    self._session.pause_writing()
AttributeError: 'NoneType' object has no attribute 'pause_writing'
------------------------------------------------------------
2021-11-15 21:12:53,066 DEBUG: Version info for developers:
DVC version: 2.8.4.dev20+g28dd39a1
---------------------------------
Platform: Python 3.8.8 on Linux-5.4.0-77-generic-x86_64-with-glibc2.10
Supports:
	azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.6.0),
	gdrive (pydrive2 = 1.10.0),
	gs (gcsfs = 2021.11.0),
	hdfs (fsspec = 2021.11.0, pyarrow = 4.0.1),
	webhdfs (fsspec = 2021.11.0),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.11.0, boto3 = 1.17.106),
	ssh (sshfs = 2021.11.0),
	oss (ossfs = 2021.8.0),
	webdav (webdav4 = 0.9.3),
	webdavs (webdav4 = 0.9.3)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-11-15 21:12:53,074 DEBUG: Analytics is enabled.
2021-11-15 21:12:53,134 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpdooeuvhh']'
2021-11-15 21:12:53,135 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpdooeuvhh']'
$ dvc doctor
DVC version: 2.8.4.dev20+g28dd39a1
---------------------------------
Platform: Python 3.8.8 on Linux-5.4.0-77-generic-x86_64-with-glibc2.10
Supports:
	azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.6.0),
	gdrive (pydrive2 = 1.10.0),
	gs (gcsfs = 2021.11.0),
	hdfs (fsspec = 2021.11.0, pyarrow = 4.0.1),
	webhdfs (fsspec = 2021.11.0),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.11.0, boto3 = 1.17.106),
	ssh (sshfs = 2021.11.0),
	oss (ossfs = 2021.8.0),
	webdav (webdav4 = 0.9.3),
	webdavs (webdav4 = 0.9.3)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

pmrowla · 2021-11-16T04:52:42Z

@karajan1001 please try the latest version of the PR and see if it resolves the push issue

efiop · 2021-11-16T15:42:24Z

Works with my keys, at least.

dberenbaum · 2021-11-16T18:28:58Z

It works for me only if I have an empty passphrase. With a passphrase set, I get:

$ dvc exp push -v --no-cache dberenbaum exp-bb09c
2021-11-16 13:24:52,529 DEBUG: git push experiment 'refs/exps/06/eed83b30c1e3d6cd7be76c0965d7e0ea56439e/exp-bb09c' -> 'dberenbaum'
2021-11-16 13:24:52,569 ERROR: unexpected error - Passphrase must be specified to import encrypted private keys
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/Users/dave/Code/dvc/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/Users/dave/Code/dvc/dvc/command/experiments.py", line 731, in run
    self.repo.experiments.push(
  File "/Users/dave/Code/dvc/dvc/repo/experiments/__init__.py", line 1001, in push
    return push(self.repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/scm_context.py", line 14, in run
    return method(repo, *args, **kw)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/push.py", line 40, in push
    repo.scm.push_refspec(
  File "/Users/dave/Code/dvc/dvc/scm/git/__init__.py", line 296, in _backend_func
    return func(*args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/scm/git/backend/dulwich/__init__.py", line 452, in push_refspec
    client.send_pack(
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/dulwich/client.py", line 917, in send_pack
    proto, unused_can_read, stderr = self._connect(b"receive-pack", path)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/dulwich/client.py", line 1661, in _connect
    con = self.ssh_vendor.run_command(
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/Users/dave/Code/dvc/dvc/scm/git/backend/dulwich/asyncssh_vendor.py", line 131, in _run_command
    conn = await asyncssh.connect(
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/connection.py", line 6850, in connect
    options = SSHClientConnectionOptions(options, config=config, host=host,
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/connection.py", line 5596, in __init__
    super().__init__(options=options, last_config=last_config, **kwargs)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/misc.py", line 268, in __init__
    self.prepare(**self.kwargs)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/connection.py", line 6307, in prepare
    self.client_keys = load_keypairs(client_keys, passphrase,
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 3152, in load_keypairs
    key, certs = read_private_key_and_certs(key, passphrase)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 2969, in read_private_key_and_certs
    key, cert = import_private_key_and_certs(read_file(filename), passphrase)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 2859, in import_private_key_and_certs
    key, end = _decode_private(data, passphrase)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 2519, in _decode_private
    key = _decode_pem_private(pem_name, headers, data, passphrase)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 2433, in _decode_pem_private
    return _decode_openssh_private(data, passphrase)
  File "/Users/dave/miniforge3/envs/apple_tf/lib/python3.8/site-packages/asyncssh/public_key.py", line 2276, in _decode_openssh_private
    raise KeyImportError('Passphrase must be specified to import '
asyncssh.public_key.KeyImportError: Passphrase must be specified to import encrypted private keys
------------------------------------------------------------
2021-11-16 13:24:53,030 DEBUG: Version info for developers:
DVC version: 2.8.4.dev12+gd668cbeb.d20211111
---------------------------------
Platform: Python 3.8.8 on macOS-12.0.1-arm64-arm-64bit
Supports:
        azure (adlfs = 2021.9.1, knack = 0.9.0, azure-identity = 1.7.1),
        gdrive (pydrive2 = 1.10.0),
        gs (gcsfs = 2021.11.0),
        webhdfs (fsspec = 2021.11.0),
        http (aiohttp = 3.7.4, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.11.0, boto3 = 1.19.8),
        ssh (sshfs = 2021.11.0),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.3),
        webdavs (webdav4 = 0.9.3)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-11-16 13:24:53,031 DEBUG: Analytics is disabled.

pmrowla · 2021-11-17T00:01:04Z

@dberenbaum are you normally prompted for your passphrase when you git push (or exp push in older releases)? Also can you pip freeze|grep asyncssh

asyncssh should already be using ssh-agent to get the passphrase w/o a prompt (assuming you have already added it to your local/system keychain). It looks like there was a bug here that was fixed in 2.7.1 (ronf/asyncssh#394) so if you're running 2.7.0 that might be the reason. We'll probably need to bump the minimum required version in DVC and sshfs so that we get that bugfix (it's currently `>=2.7,<2.8)

dberenbaum · 2021-11-17T02:19:37Z

@dberenbaum are you normally prompted for your passphrase when you git push (or exp push in older releases)? Also can you pip freeze|grep asyncssh

No. git push works without any prompting and so does exp push for dvc<=2.8.1.

asyncssh should already be using ssh-agent to get the passphrase w/o a prompt (assuming you have already added it to your local/system keychain). It looks like there was a bug here that was fixed in 2.7.1 (ronf/asyncssh#394) so if you're running 2.7.0 that might be the reason. We'll probably need to bump the minimum required version in DVC and sshfs so that we get that bugfix (it's currently `>=2.7,<2.8)

$ pip freeze | grep asynssh
asyncssh==2.7.2

My ~/.ssh/config:

Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519

I also tried with an rsa key and got the same error.

karajan1001 · 2021-11-17T02:27:38Z

@karajan1001 please try the latest version of the PR and see if it resolves the push issue

Works for me now.

 $ dvc exp push origin --no-cache name
Pushed experiment 'name'to Git remote 'origin'.
To push cached outputs for this experiment to DVC remote storage,re-run this command without '--no-cache'.
$ dvc exp push origin name
ERROR: failed to transfer 'md5 (888d1c818ea36e6772a17d45223c1daff7333052): dc5bb2d3f7e0906ece5a962866532f2b' - Server disconnected
ERROR: failed to transfer 'md5 (888d1c818ea36e6772a17d45223c1daff7333052): 3c8f026df7c3973bcec8e9f945026eb2' - Server disconnected
ERROR: failed to transfer 'md5 (57e8d6f48431025a93410b21ba4c8943115fcc26): 36e8ae5b35c7780f8f55899ffb96ab80' - Server disconnected
ERROR: 3 files failed to upload

pmrowla · 2021-11-17T05:37:25Z

@dberenbaum the passphrase issue should be resolved in the latest version of this PR

- vendor should use ssh-agent or default IdentityFile(s) when no SSH key has been provided by the user - encrypted (fallback) keys will always be passed to ssh-agent

- allow mismatched RSA signature algo names in initial SSH_MSG_USERAUTH_PK_OK response - unit test overridden asyncssh publickey auth compatibility with github ssh server

pmrowla · 2021-11-17T06:06:48Z

This is ready for review but shouldn't get merged until there's a new sshfs release w/the updated asyncssh dependency version

efiop · 2021-11-17T10:25:53Z

sshfs 2021.11.1 is out

dberenbaum · 2021-11-17T16:37:35Z

Nice! It works now, thanks.

Comment on rebasing: I went to look for what was changed to fix my earlier issues, but I can't tell anymore. Not a big deal.

pmrowla · 2021-11-18T00:38:02Z

Nice! It works now, thanks.

Comment on rebasing: I went to look for what was changed to fix my earlier issues, but I can't tell anymore. Not a big deal.

It was originally a separate commit in the PR, but it was a small one-line change that fit better into aca229a so I squashed it in the final PR changeset (the specific change is the inclusion of ignore_encrypted=... which forces asyncssh to pass encrypted keys into the ssh-agent when possible instead of handling them itself

pmrowla added the bugfix fixes bug label Nov 15, 2021

pmrowla requested a review from a team as a code owner November 15, 2021 11:03

pmrowla self-assigned this Nov 15, 2021

pmrowla requested a review from skshetry November 15, 2021 11:03

pmrowla commented Nov 15, 2021

View reviewed changes

pmrowla force-pushed the asyncssh-github branch from 446f212 to 5b7f2d3 Compare November 16, 2021 04:51

pmrowla mentioned this pull request Nov 17, 2021

setup: bump asyncssh to 2.7.1 fsspec/sshfs#12

Merged

pmrowla force-pushed the asyncssh-github branch from 7ce23db to fc6229b Compare November 17, 2021 06:00

pmrowla changed the title ~~[WIP] dulwich: fallback to default SSH keys in asyncssh vendor~~ dulwich: fallback to default SSH keys in asyncssh vendor Nov 17, 2021

pmrowla added the A: experiments Related to dvc exp label Nov 17, 2021

pmrowla added 3 commits November 17, 2021 15:03

dulwich: fallback to default client config in asyncssh vendor

aca229a

- vendor should use ssh-agent or default IdentityFile(s) when no SSH key has been provided by the user - encrypted (fallback) keys will always be passed to ssh-agent

dulwich: workaround github SSH server in asyncssh vendor

366c96b

- allow mismatched RSA signature algo names in initial SSH_MSG_USERAUTH_PK_OK response - unit test overridden asyncssh publickey auth compatibility with github ssh server

dulwich: flush stdin channel on asyncssh vendor write

6c3adba

pmrowla force-pushed the asyncssh-github branch from fc6229b to 6c3adba Compare November 17, 2021 06:04

efiop approved these changes Nov 17, 2021

View reviewed changes

efiop merged commit 28f5793 into iterative:master Nov 17, 2021

pmrowla deleted the asyncssh-github branch November 17, 2021 10:54

iesahin mentioned this pull request Dec 8, 2021

check documentation for dvc 2.9 release iterative/dvc.org#3074

Closed

21 tasks

skshetry mentioned this pull request Feb 6, 2025

remove asyncssh patching iterative/scmrepo#404

Merged

dulwich: fallback to default SSH keys in asyncssh vendor #6983

dulwich: fallback to default SSH keys in asyncssh vendor #6983

Uh oh!

Conversation

pmrowla commented Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmrowla commented Nov 15, 2021

Uh oh!

pmrowla Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karajan1001 commented Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmrowla commented Nov 16, 2021

Uh oh!

efiop commented Nov 16, 2021

Uh oh!

dberenbaum commented Nov 16, 2021

Uh oh!

pmrowla commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dberenbaum commented Nov 17, 2021

Uh oh!

karajan1001 commented Nov 17, 2021

Uh oh!

pmrowla commented Nov 17, 2021

Uh oh!

pmrowla commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efiop commented Nov 17, 2021

Uh oh!

dberenbaum commented Nov 17, 2021

Uh oh!

pmrowla commented Nov 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pmrowla commented Nov 15, 2021 •

edited

Loading

pmrowla Nov 15, 2021 •

edited

Loading

karajan1001 commented Nov 15, 2021 •

edited

Loading

pmrowla commented Nov 17, 2021 •

edited

Loading

pmrowla commented Nov 17, 2021 •

edited

Loading

pmrowla commented Nov 18, 2021 •

edited

Loading