Homebrew `dvc` version is missing `invoke` dependency

[Pseudomanifold]

Bug Report

Upon installing dvc on Mac OS X Catalina 10.15.6 (19G2021) using Homebrew, the dvc push command fails with the message:

dvc ERROR: unexpected error - No module named 'invoke'

This can be fixed by invoking (no pun intended) pip3 install invoke in the virtual environment used by dcv. More precisely, the following workaround solves the issue:

$ source /usr/local/Cellar/dvc/1.7.2/libexec/bin/activate
$ pip3 install invoke

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.7.2 (brew)
---------------------------------
Platform: Python 3.8.5 on macOS-10.15.6-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Repo: dvc, git

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[efiop]

Hi @Pseudomanifold !

Could you please post a full log for dvc push -v with that problem? What remote are you using?

[Pseudomanifold]

Of course!

I am using an SSH remote. Here's the full output:

~/Projects/Hisper % dvc push -v
2020-09-22 11:21:28,674 DEBUG: Check for update is enabled.
2020-09-22 11:21:28,677 DEBUG: fetched: [(3,)]
2020-09-22 11:21:28,681 DEBUG: Assuming '/Users/rieckb/Projects/Hisper/.dvc/cache/ea/9407b8388f85e2f731b7e2e08c2183.dir' is unchanged since it is read-only
2020-09-22 11:21:28,683 DEBUG: Assuming '/Users/rieckb/Projects/Hisper/.dvc/cache/ea/9407b8388f85e2f731b7e2e08c2183.dir' is unchanged since it is read-only
2020-09-22 11:21:28,701 DEBUG: Assuming '/Users/rieckb/Projects/Hisper/.dvc/cache/f1/9aa2457957d66c2ee5e101871f402b.dir' is unchanged since it is read-only
2020-09-22 11:21:28,702 DEBUG: Assuming '/Users/rieckb/Projects/Hisper/.dvc/cache/f1/9aa2457957d66c2ee5e101871f402b.dir' is unchanged since it is read-only
2020-09-22 11:21:28,768 DEBUG: fetched: [(11315,)]
2020-09-22 11:21:28,774 ERROR: unexpected error - No module named 'invoke'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/main.py", line 76, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/command/data_sync.py", line 50, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/repo/__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/repo/push.py", line 35, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/data_cloud.py", line 63, in push
    remote = self.get_remote(remote, "push")
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/data_cloud.py", line 30, in get_remote
    return self._init_remote(name)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/data_cloud.py", line 48, in _init_remote
    return get_remote(self.repo, name=name)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/remote/__init__.py", line 8, in get_remote
    tree = get_cloud_tree(repo, **kwargs)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/tree/__init__.py", line 88, in get_cloud_tree
    return _get_tree(remote_conf)(repo, remote_conf)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/tree/ssh/__init__.py", line 54, in __init__
    user_ssh_config = self._load_user_ssh_config(parsed.hostname)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/dvc/tree/ssh/__init__.py", line 111, in _load_user_ssh_config
    user_ssh_config = ssh_config.lookup(hostname)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/paramiko/config.py", line 223, in lookup
    options = self._lookup(hostname=hostname)
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/paramiko/config.py", line 250, in _lookup
    or self._does_match(
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/paramiko/config.py", line 387, in _does_match
    raise invoke_import_error
  File "/usr/local/Cellar/dvc/1.7.2/libexec/lib/python3.8/site-packages/paramiko/config.py", line 36, in <module>
    import invoke
ModuleNotFoundError: No module named 'invoke'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
DVC version: 1.7.2 (brew)
---------------------------------
Platform: Python 3.8.5 on macOS-10.15.6-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Repo: dvc, git
2020-09-22 11:21:29,327 DEBUG: Analytics is enabled.
2020-09-22 11:21:29,548 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/27/ynjbd_pj1rl5rv8jky0_7vgr0000gq/T/tmprgkuv1ax']'
2020-09-22 11:21:29,550 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/27/ynjbd_pj1rl5rv8jky0_7vgr0000gq/T/tmprgkuv1ax']'

[efiop]

Thanks! So looks like we lost paramiko's dependency somewhere. We'll take a closer look today. Thanks for the feedback! 🙏

[Pseudomanifold]

You are welcome; thank you very much for the quick response and for the awesome documentation! Let me know if I should test anything else.

[efiop]

Ok, so looks like paramiko has recently added support for Match paramiko/paramiko#717 , but it requires invoke extra dependency. Looks like we can simply require paramiko[invoke] by default, but I'm a bit worried about whether or not that option works well right now, but looks promising and looks like we just have to try it.

[efiop]

@Pseudomanifold Btw, are you using Match in your ssh config?