Skip to content

import: failing with SSH key-based authentication from a GitHub repo #9737

@wjn0

Description

@wjn0

Bug Report

Description

I have a private github repo username/repo.git created from following the data registry tutorial on my own data. I am trying to import a file from it like dvc import git@github.com:username/repo.git filename and it is failing on an ssh error.

Reproduce

  1. Create directory tmprepo and cd tmprepo
  2. git init
  3. dvc init
  4. dvc import git@github.com:username/repo.git filename I have also tried adding ssh://... protocol indicator and a few other modifications, no dice)
Output with `-v`:
2023-07-14 10:37:43,026 DEBUG: v3.5.1 (pip), CPython 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
2023-07-14 10:37:43,027 DEBUG: command: /home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/bin/dvc import -v ssh://git@github.com/githubusername/projectname_data.git raw -o data/raw
2023-07-14 10:37:43,529 DEBUG: Removing output 'data/raw' of stage: 'data/raw.dvc'.
2023-07-14 10:37:43,530 DEBUG: Removing '/storage/dhi_work/username/projectname_preprocessing/data/raw'
Importing 'raw (ssh://git@github.com/githubusername/projectname_data.git)' -> 'data/raw'
2023-07-14 10:37:43,547 DEBUG: Computed stage: 'data/raw.dvc' md5: 'a5b888cce3b0c91f50f9272fecd11db3'
2023-07-14 10:37:43,547 DEBUG: 'md5' of stage: 'data/raw.dvc' changed.
2023-07-14 10:37:43,548 DEBUG: Creating external repo ssh://git@github.com/githubusername/projectname_data.git@None
2023-07-14 10:37:43,548 DEBUG: erepo: git clone 'ssh://git@github.com/githubusername/projectname_data.git' to a temporary dir
2023-07-14 10:37:43,877 ERROR: failed to import 'raw' - SCM error: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone': Authentication failed for: 'git@github.com:22': Permission denied
Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 204, in _run_command
    conn = await asyncssh.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 8093, in connect
    return await asyncio.wait_for(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lmod/packages/python/3.11.3/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 440, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 220, in clone
    repo = clone_from()
           ^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/porcelain.py", line 514, in clone
    return client.clone(
           ^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 703, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 781, in fetch
    result = self.fetch_pack(
             ^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1097, in fetch_pack
    proto, can_read, stderr = self._connect(b"upload-pack", path)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1746, in _connect
    con = self.ssh_vendor.run_command(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 217, in _run_command
    raise AuthError(f"{username}@{host}:{port or 22}") from exc
scmrepo.exceptions.AuthError: Authentication failed for: 'git@github.com:22'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 160, in clone
    git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/__init__.py", line 142, in clone
    backend.clone(url, to_path, **kwargs)
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 225, in clone
    raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/commands/imp.py", line 17, in run
    self.repo.imp(
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp.py", line 6, in imp
    return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp_url.py", line 85, in imp_url
    stage.run(jobs=jobs, no_download=no_download)
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
           ^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 600, in run
    self._sync_import(dry, force, kwargs.get("jobs", None), no_download)
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
           ^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 637, in _sync_import
    sync_import(self, dry, force, jobs, no_download)
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/imports.py", line 57, in sync_import
    stage.save_deps()
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 491, in save_deps
    dep.save()
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/dependency/repo.py", line 58, in save
    rev = self.fs.repo.get_rev()
          ^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 427, in repo
    return self.fs.repo
           ^^^^^^^
  File "/lmod/packages/python/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 416, in fs
    return _DVCFileSystem(**self.fs_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 128, in __init__
    repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 188, in _make_repo
    with Repo.open(uninitialized=True, **kwargs) as repo:
  File "/lmod/packages/python/3.11.3/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 30, in _external_repo
    path = _cached_clone(url, rev, for_write=for_write)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 154, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/flow.py", line 246, in wrap_with
    return call()
           ^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 222, in _clone_default_branch
    git = clone(url, clone_path)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 165, in clone
    raise CloneError("SCM error") from exc
dvc.scm.CloneError: SCM error

2023-07-14 10:37:43,889 DEBUG: Analytics is enabled.
2023-07-14 10:37:43,950 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']'
2023-07-14 10:37:43,952 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']'

Expected

I expect to be prompted for my SSH key password and then I expect filename to be imported to the repo.

Environment information

Initially encountered under Poetry but confirmed it occurs outside of Poetry as well. Tried downgrading to dvc 2.9.5 based on another github bug report but failed on an import so I think that version of the package is just too old for me to try. I tried adding dvc[ssh] and this did not work (same error).

Output of dvc doctor:

$ poetry run dvc doctor
DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
Subprojects:
        dvc_data = 2.5.0
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.4
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.4.1)
Config:
        Global: /home/username/.config/dvc
        System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: nfs on nfs:/nfs_path
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/435bb81e569bac9d0867f3b2c02bd890

Additional Information (if any):

The remote is local for now but given that it fails before prompting for my SSH key password I don't think that's relevant.

Metadata

Metadata

Assignees

Labels

A: data-syncRelated to dvc get/fetch/import/pull/pushgitRelated to git and git backendsp1-importantImportant, aka current backlog of things to do

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions