DVC Pull Ignores identityfile in the SSH config.

[Erotemic]

Bug Report

Potentially related to #1965 #6225 and this may also be a Paramiko bug based on what I've readl.

The underlying issue is that a dvc pull does not seem to support the identityfile in my $HOME/.ssh/config.

My ssh config has a entry that looks like this:

Host bigbox bigbox.kitware.com
    HostName bigbox.kitware.com
    Port 22
    User jon.crall
    identityfile ~/.ssh/id_kitware_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes

My .dvc/cache looks like:

[cache]
    type = "reflink,symlink,hardlink,copy"
    shared = group
    protected = true
[core]
    remote = bigbox
    check_update = False
['remote "bigbox"']
    url = ssh://bigbox.kitware.com/data/dvc-caches/crall_dvc
    port = 22

And I'm able to ssh bigbox just fine. But when I attempted to run dvc pull -r bigbox I got a permission error.

Traceback (most recent call last):
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/dvc/objects/db/base.py", line 443, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 93, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/dvc/fs/ssh.py", line 114, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/fsspec/spec.py", line 75, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/sshfs/spec.py", line 61, in __init__
    self._client, self._pool = self.connect(
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/fsspec/asyn.py", line 88, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/fsspec/asyn.py", line 69, in sync
    raise result[0]
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 491, in wait_for
    return fut.result()
  File "/home/joncrall/.pyenv/versions/3.8.6/envs/pyenv3.8.6/lib/python3.8/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied

I ended up digging and printing out some info in the objects/db/base.py file and printed:

  0% Querying remote cache|                                                                                                                              |0/1 [00:00<?,     ?file/s]path_infos = [URLInfo: 'ssh://bigbox.kitware.com/data/dvc-caches/crall_dvc/97/12707710626642715101298436624963.dir']
path_info = URLInfo: 'ssh://bigbox.kitware.com/data/dvc-caches/crall_dvc/97/12707710626642715101298436624963.dir'
self.fs = <dvc.fs.ssh.SSHFileSystem object at 0x7fc71425b790>

I stepped into this function with a breakpoint and looked at self.fs.__dict__ and saw:

In [5]: self.__dict__
Out[5]: 
{'jobs': 128,
 'hash_jobs': 4,
 '_config': {'host': 'bigbox.kitware.com',
  'port': 22,
  'tmp_dir': '/media/joncrall/raid/home/joncrall/data/dvc-repos/crall_dvc/.dvc/tmp'},
 'fs_args': {'skip_instance_cache': True,
  'host': 'bigbox.kitware.com',
  'username': 'jon.crall',
  'port': 22,
  'password': None,
  'client_keys': ['/home/joncrall/.ssh/id_personal_ed25519'],
  'timeout': 1800,
  'encryption_algs': ['aes128-gcm@openssh.com',
   'aes256-ctr',
   'aes192-ctr',
   'aes128-ctr'],
  'compression_algs': None,
  'gss_auth': False,
  'agent_forwarding': True,
  'proxy_command': None,
  'known_hosts': None},
 'CAN_TRAVERSE': True}

The main issue being that client_keys was set to /home/joncrall/.ssh/id_personal_ed25519, which is incorrect based on the config.

I was able to work around this by adding my id_personal_ed25519.pub to authorized keys on bigbox, but this is not ideal.

It might be useful to add some debugging info when -v is passed to DVC that prints out some of this SSHFileSystem information, because if I saw this wrong identity file two weeks ago, it would have saved me a ton of time.

[pmrowla]

Can you post the output of dvc doctor? It looks like you are probably using the latest version based on the traceback, but it's helpful for us to know for sure. Also, DVC no longer uses Paramiko (the underlying backend is now asyncssh).

[Erotemic]

(pyenv3.8.6) joncrall@namek:~$ dvc doctor
DVC version: 2.6.4 (pip)
---------------------------------
Platform: Python 3.8.6 on Linux-5.4.0-80-generic-x86_64-with-glibc2.2.5
Supports:
	azure (adlfs = 2021.7.1, knack = 0.8.2, azure-identity = 1.6.0),
	gdrive (pydrive2 = 1.9.1),
	gs (gcsfs = 2021.7.0),
	hdfs (pyarrow = 5.0.0),
	webhdfs (hdfs = 2.5.8),
	http (requests = 2.25.1),
	https (requests = 2.25.1),
	s3 (s3fs = 2021.7.0, boto3 = 1.17.49),
	ssh (sshfs = 2021.7.1),
	oss (ossfs = 2021.7.5),
	webdav (webdav4 = 0.9.0),
	webdavs (webdav4 = 0.9.0)

Anything out of the ordinary? I did attempt a pip install dvc -U before I reported.

[pmrowla]

CC @isidentical

[isidentical]

The main issue being that client_keys was set to /home/joncrall/.ssh/id_personal_ed25519, which is incorrect based on the config.

identityfile ~/.ssh/id_kitware_ed25519

Am I missing something, since the client_keys is identical to your IdentityFile setting?

[pmrowla]

Looks like the IdentityFile entry is id_kitware_... but asyncssh is using id_personal_...

[isidentical]

Now I see the difference, thanks @pmrowla!

@Erotemic I assume that you have another config that overrides the setting, can you also share that?

I tried a couple variations;

Host example.com
   User ubuntu
   HostName 1.2.3.4
   Port 1234
   IdentityFile ~/.ssh/not_default.key

Host kitware.com
    identityfile ~/.ssh/id_personal_ed25519

Host bigbox bigbox.kitware.com
    HostName bigbox.kitware.com
    Port 22
    User jon.crall
    identityfile ~/.ssh/id_kitware_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes

and in every time, it successfully parsed the correct identityfile;

>>> from sshfs.config import parse_config
>>> 
>>> print(parse_config(host='bigbox.kitware.com')._options)
{'Hostname': 'bigbox.kitware.com', 'Port': 22, 'User': 'jon.crall', 'IdentityFile': ['~/.ssh/id_kitware_ed25519'], 'ForwardX11Trusted': True}

[Erotemic]

I don't have another config file that I know of (but if you know how I could find a potential override if one exists, let me know, I only see the one config).

However, I think I see the problem. At the top of the config I have a separate default IdentityFile entry at the top of the ssh config. I think that is what is causing the issue. (The rest of the lines in the SSH file are all Host entries that point at different machines).

IdentityFile ~/.ssh/id_personal_ed25519

Host bigbox bigbox.kitware.com
    HostName bigbox.kitware.com
    Port 22
    User jon.crall
    identityfile ~/.ssh/id_kitware_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes

Using your example I see this:

>>> from sshfs.config import parse_config
>>> 
>>> print(parse_config(host='bigbox.kitware.com')._options)

which results in

{'IdentityFile': ['~/.ssh/id_personal_ed25519',
  '~/.ssh/id_kitware_ed25519'],
 'Hostname': 'bigbox.kitware.com',
 'Port': 22,
 'User': 'jon.crall',
 'ForwardX11Trusted': True}

Both of the keys show up in the IdentityFile list! I'm wondering if there is something in asyncssh or dvc that is using config['IdentityFile'][0] and ignoring the second entry?

[isidentical]

Yes! That seems like the issue

dvc/dvc/fs/ssh.py

Lines 71 to 75 in e5cf9b2

	if user_ssh_config.get("IdentityFile"):
	config.setdefault(
	"keyfile", first(user_ssh_config.get("IdentityFile"))
	)

I am not rally sure about the reason why we choose this behavior (cc: @efiop), and it might simply be something that stuck during the initial times of the SSH (this behavior is kept since paramiko AFAIK);

dvc/dvc/fs/ssh/__init__.py

Lines 120 to 121 in 5e4de84

	def _try_get_ssh_config_keyfile(user_ssh_config):
	return first(user_ssh_config.get("identityfile") or ())

[Erotemic]

So with ssh itself, the behavior seems to be that it will try each identity file until one works. I think this issue has some potential action items that could improve DVC:

• Write out the user_ssh_config as a DEBUG level log so this is clear when -v is given (I think this is necessary)
• If a permission error occurs try the next identity file if another one is available (not sure how contentious this is, but this change would make it work more like ssh itself).

[efiop]

@isidentical Yeah, it just stuck around since #1965 We could totally consider changing it, even at the cost of backward compatibility, since this is a rather advanced feature and it is already broken. Up to you guys.