-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Bug Report
Description
I have a private github repo username/repo.git created from following the data registry tutorial on my own data. I am trying to import a file from it like dvc import git@github.com:username/repo.git filename and it is failing on an ssh error.
Reproduce
- Create directory
tmprepoandcd tmprepo git initdvc initdvc import git@github.com:username/repo.git filenameI have also tried addingssh://...protocol indicator and a few other modifications, no dice)
Output with `-v`:
2023-07-14 10:37:43,026 DEBUG: v3.5.1 (pip), CPython 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
2023-07-14 10:37:43,027 DEBUG: command: /home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/bin/dvc import -v ssh://git@github.com/githubusername/projectname_data.git raw -o data/raw
2023-07-14 10:37:43,529 DEBUG: Removing output 'data/raw' of stage: 'data/raw.dvc'.
2023-07-14 10:37:43,530 DEBUG: Removing '/storage/dhi_work/username/projectname_preprocessing/data/raw'
Importing 'raw (ssh://git@github.com/githubusername/projectname_data.git)' -> 'data/raw'
2023-07-14 10:37:43,547 DEBUG: Computed stage: 'data/raw.dvc' md5: 'a5b888cce3b0c91f50f9272fecd11db3'
2023-07-14 10:37:43,547 DEBUG: 'md5' of stage: 'data/raw.dvc' changed.
2023-07-14 10:37:43,548 DEBUG: Creating external repo ssh://git@github.com/githubusername/projectname_data.git@None
2023-07-14 10:37:43,548 DEBUG: erepo: git clone 'ssh://git@github.com/githubusername/projectname_data.git' to a temporary dir
2023-07-14 10:37:43,877 ERROR: failed to import 'raw' - SCM error: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone': Authentication failed for: 'git@github.com:22': Permission denied
Traceback (most recent call last):
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 204, in _run_command
conn = await asyncssh.connect(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 8093, in connect
return await asyncio.wait_for(
^^^^^^^^^^^^^^^^^^^^^^^
File "/lmod/packages/python/3.11.3/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
return await fut
^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 440, in _connect
await options.waiter
asyncssh.misc.PermissionDenied: Permission denied
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 220, in clone
repo = clone_from()
^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/porcelain.py", line 514, in clone
return client.clone(
^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 703, in clone
result = self.fetch(path, target, progress=progress, depth=depth)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 781, in fetch
result = self.fetch_pack(
^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1097, in fetch_pack
proto, can_read, stderr = self._connect(b"upload-pack", path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1746, in _connect
con = self.ssh_vendor.run_command(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 121, in wrapper
return sync(self.loop, func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 106, in sync
raise return_result
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 61, in _runner
result[0] = await coro
^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 217, in _run_command
raise AuthError(f"{username}@{host}:{port or 22}") from exc
scmrepo.exceptions.AuthError: Authentication failed for: 'git@github.com:22'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 160, in clone
git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/__init__.py", line 142, in clone
backend.clone(url, to_path, **kwargs)
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 225, in clone
raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/commands/imp.py", line 17, in run
self.repo.imp(
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp.py", line 6, in imp
return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/__init__.py", line 64, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 151, in run
return method(repo, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp_url.py", line 85, in imp_url
stage.run(jobs=jobs, no_download=no_download)
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
return call()
^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 600, in run
self._sync_import(dry, force, kwargs.get("jobs", None), no_download)
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
return call()
^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 637, in _sync_import
sync_import(self, dry, force, jobs, no_download)
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/imports.py", line 57, in sync_import
stage.save_deps()
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 491, in save_deps
dep.save()
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/dependency/repo.py", line 58, in save
rev = self.fs.repo.get_rev()
^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 427, in repo
return self.fs.repo
^^^^^^^
File "/lmod/packages/python/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 416, in fs
return _DVCFileSystem(**self.fs_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
obj = super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 128, in __init__
repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 188, in _make_repo
with Repo.open(uninitialized=True, **kwargs) as repo:
File "/lmod/packages/python/3.11.3/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 30, in _external_repo
path = _cached_clone(url, rev, for_write=for_write)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 154, in _cached_clone
clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/flow.py", line 246, in wrap_with
return call()
^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 222, in _clone_default_branch
git = clone(url, clone_path)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 165, in clone
raise CloneError("SCM error") from exc
dvc.scm.CloneError: SCM error
2023-07-14 10:37:43,889 DEBUG: Analytics is enabled.
2023-07-14 10:37:43,950 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']'
2023-07-14 10:37:43,952 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']'
Expected
I expect to be prompted for my SSH key password and then I expect filename to be imported to the repo.
Environment information
Initially encountered under Poetry but confirmed it occurs outside of Poetry as well. Tried downgrading to dvc 2.9.5 based on another github bug report but failed on an import so I think that version of the package is just too old for me to try. I tried adding dvc[ssh] and this did not work (same error).
Output of dvc doctor:
$ poetry run dvc doctor
DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
Subprojects:
dvc_data = 2.5.0
dvc_objects = 0.23.0
dvc_render = 0.5.3
dvc_task = 0.3.0
scmrepo = 1.0.4
Supports:
http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
ssh (sshfs = 2023.4.1)
Config:
Global: /home/username/.config/dvc
System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: nfs on nfs:/nfs_path
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/435bb81e569bac9d0867f3b2c02bd890Additional Information (if any):
The remote is local for now but given that it fails before prompting for my SSH key password I don't think that's relevant.