`get`: fails to clone because "no valid credentials provided"

[guyrosin]

Bug Report

Description

When executing dvc get or dvc update, a "failed to clone repo" error appears, which originates from a "dulwich.client.HTTPUnauthorized: No valid credentials provided" error.
This started happening without any clear reason, after several weeks of working with DVC with no problem.

Related issues: jelmer/dulwich#882, python-poetry/poetry#5428

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.9.12 on macOS-12.2.1-x86_64-i386-64bit
Supports:
        hdfs (fsspec = 2022.3.0, pyarrow = 7.0.0),
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

Additional Information (if any):

$ dvc get https://dagshub.com/aviv/data-repo README.md -v
2022-05-01 16:37:20,420 DEBUG: Creating external repo https://dagshub.com/aviv/data-repo@None
2022-05-01 16:37:20,420 DEBUG: erepo: git clone 'https://dagshub.com/aviv/data-repo' to a temporary dir
2022-05-01 16:37:20,888 DEBUG: Removing '/Users/guyrosin/code/playground/test/.B66n2XDC3RLgwwhXr8AoUr'                                                                           
2022-05-01 16:37:20,888 ERROR: failed to get 'README.md' from 'https://dagshub.com/aviv/data-repo' - Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmpv7y9y9z5dvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 193, in clone
    repo = clone_from()
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/porcelain.py", line 443, in clone
    return client.clone(
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 535, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 601, in fetch
    result = self.fetch_pack(
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 2047, in fetch_pack
    refs, server_capabilities, url = self._discover_references(
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 1908, in _discover_references
    resp, read = self._http_request(url, headers, allow_compression=True)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 2189, in _http_request
    raise HTTPUnauthorized(resp.getheader("WWW-Authenticate"), url)
dulwich.client.HTTPUnauthorized: No valid credentials provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/scm.py", line 126, in clone
    git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 143, in clone
    backend.clone(url, to_path, **kwargs)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 196, in clone
    raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmpv7y9y9z5dvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/commands/get.py", line 39, in _get_file_from_repo
    Repo.get(
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/repo/get.py", line 49, in get
    with external_repo(
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 39, in external_repo
    path = _cached_clone(url, rev, for_write=for_write)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 165, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/flow.py", line 274, in wrap_with
    return call()
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 235, in _clone_default_branch
    git = clone(url, clone_path)
  File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/scm.py", line 131, in clone
    raise CloneError(str(exc))
dvc.scm.CloneError: Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmpv7y9y9z5dvc-clone'
------------------------------------------------------------

[dtrifiro]

Hi,
I see that https://dagshub.com/aviv/data-repo gives me a 404. I'm assuming this is because that is a private repo (either that or it has been deleted)

Since you're saying that this started happening recently was this changed to private (deleted?) recently?

[guyrosin]

Thanks @dtrifiro, this repo has always been private. I've been using it for several weeks already with no problem.
It seems this error started appearing after I updated DVC to v2.10! After downgrading to v2.9.3 everything seems to work.

Here's the whole debugging story FYI:
Initially I used DVC v2.10.3. I tried running dvc get and dvc import with two git URLs: https://dagshub.com/aviv/data-repo, and using an access token (https://{user}:{token}@dagshub.com/aviv/data-repo). Both didn't work.
I've downgraded DVC to v2.10.1 and it looked much better, but still weird...

• First of all, dvc get and dvc import actually worked!
• They worked only when using the URL with the user:token prefix! Otherwise I got the same error as before ("No valid credentials provided")
• After the successful dvc import, if I executed the same command again without any file changes, (almost) the same error appeared: "unexpected error - No valid credentials provided".
• But if I push to dvc and git and then run dvc import, it worked...
• Finally, I downgraded DVC to v2.9.3 (the last version I remember worked 100%), and now running dvc import without any file changes results in "'data.dvc' didn't change, skipping", as it should've been. It works even with the standard HTTPS URL (without the user:token prefix), as it should.

[dtrifiro]

Can confirm this is happening, starting from #7554