get: fails with head detached and no branches in place

[aguschin]

Bug Report

Description

With head detached and no branches available dvc get fails:

ERROR: failed to get 'data/data.xml' from '.' - unknown Git revision 'refs/remotes/origin/HEAD'

UPDATE: Skip to #6154 (comment)

Reproduce

1. git clone https://github.com/iterative/example-get-started
2. cd example-get-started
3. git checkout --detach
4. git branch -D master
5. git --no-pager branch -v
6. dvc get . data/data.xml

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.3.0 (pip)
---------------------------------
Platform: Python 3.9.5 on macOS-10.16-x86_64-i386-64bit
Supports: azure, http, https
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git

Additional Information (if any):

Interesting, that dvc pull works fine.

2021-06-11 12:45:41,751 DEBUG: Creating external repo .@None
2021-06-11 12:45:41,751 DEBUG: erepo: git clone '.' to a temporary dir
2021-06-11 12:45:41,966 DEBUG: Removing '/Users/aguschin/Git/iterative/example-get-started2/.gqZZrecSgVYzPDv8JZmpCu'                                          
2021-06-11 12:45:41,966 ERROR: failed to get 'data/data.xml' from '.' - unknown Git revision 'refs/remotes/origin/HEAD'
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/command/get.py", line 37, in _get_file_from_repo
    Repo.get(
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/repo/get.py", line 50, in get
    with external_repo(
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/external_repo.py", line 70, in external_repo
    repo = Repo(**repo_kwargs)
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/repo/__init__.py", line 155, in __init__
    self.root_dir, self.dvc_dir, self.tmp_dir = self._get_repo_dirs(
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/repo/__init__.py", line 103, in _get_repo_dirs
    fs = scm.get_fs(rev) if isinstance(scm, Git) and rev else None
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/scm/git/__init__.py", line 353, in get_fs
    resolved = self.resolve_rev(rev)
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/scm/git/__init__.py", line 400, in resolve_rev
    return self._resolve_rev(rev)
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/scm/git/__init__.py", line 343, in _backend_func
    return func(*args, **kwargs)
  File "/Users/aguschin/.local/share/virtualenvs/yolov5-y85dc7P1/lib/python3.9/site-packages/dvc/scm/git/backend/pygit2.py", line 217, in resolve_rev
    raise RevError(f"unknown Git revision '{rev}'")
dvc.scm.base.RevError: unknown Git revision 'refs/remotes/origin/HEAD'
------------------------------------------------------------
2021-06-11 12:45:41,974 DEBUG: Analytics is enabled.
2021-06-11 12:45:42,049 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/_l/xm_1fknn1479pfh9kg0ttgjc0000gq/T/tmpklkkra_a']'
2021-06-11 12:45:42,051 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/_l/xm_1fknn1479pfh9kg0ttgjc0000gq/T/tmpklkkra_a']'

Initially this happened with me in Gitlab CI -- it seems like Gitlab fetches the repo without any branches and with head detached:

# output received from debugging this in gitlab CI:

$ git --no-pager branch -v
* (HEAD detached at eae5019) eae5019 update environment

$ dvc get . model
ERROR: failed to get 'model' from '.' - unknown Git revision 'refs/remotes/origin/HEAD'                                                                       

Simple hack works, like running git checkout -b temp-branch first:

git checkout -b temp-branch
dvc get . model

[jorgeorpinel]

dvc.scm.base.RevError: unknown Git revision 'refs/remotes/origin/HEAD'

I think that get clones the source repo (even if it's .) and requires a --rev which is by default HEAD. The workaround is to use dvc get . data/data.xml --rev 00071e8 instead.

What change would you suggest here? (What would you expect?) Looks like a very rare edge case to me, but I do think that the error should be handled — maybe give message such as Please provide a valid `--rev`.

Interesting, that dvc pull works fine.

push/pull don't need Git at all. They use the metafiles present in the project (and config file).

[pmrowla]

dvc.scm.base.RevError: unknown Git revision 'refs/remotes/origin/HEAD'

I think that get clones the source repo (even if it's .) and requires a --rev which is by default HEAD. The workaround is to use dvc get . data/data.xml --rev 00071e8 instead.

I think this is a bug, the issue here is that the repo being cloned does have a HEAD, but it is detached, meaning it points to a specific revision and not to a branch. The git remote-tracking refs (like refs/remotes/origin/HEAD) are used locally to keep "copies" of refs in the cloned repo, but apparently refs/remotes/origin/HEAD isn't created when you clone a repo that is in a detached state.

This is a pretty specific edge case though, and I think cloning a repo with no branches may still cause some other issues in DVC anyways, even if we fix the origin/HEAD issue.

We maybe need to rethink how dvc get for local repos should work (and just skip the clone entirely), similar to how we should handle dvc list with local repos without cloning as well (#3590)

[pmrowla]

Should also note that if gitlab CI is always doing this specific detached fetch, you probably shouldn't be using dvc get . (with the local repo .) at all to begin with. Even when using dvc get . --rev <some_rev>, it will fail unless some_rev is a git SHA for a commit that is a parent of your detached HEAD. some_rev cannot be a branch or tag in this case, since gitlab CI is not fetching any upstream branch/tag revs, and some_rev cannot be a commit in a different branch or tag, since your local clone will only contain commits that are direct parents of your detached HEAD.

@aguschin it sounds like in gitlab CI, maybe you should always be doing

dvc get <upstream_git_url.git> --rev <revision_being_built_in_ci>

I think gitlab CI should have env vars that can be used to fill in the upstream URL and the revision. Otherwise, you are probably going to run into other issues with using the local repo .

[aguschin]

@jorgeorpinel

What change would you suggest here? (What would you expect?)

I was expecting that artefacts related to the commit eae5019 will be downloaded (in this case they are specified in dvc.lock).

push/pull don't need Git at all. They use the metafiles present in the project (and config file).

Maybe dvc get . should use the same mechanics?

@pmrowla, thank you, I think now it's the best option. For the record, in gitlab CI this will look like

dvc get $CI_PROJECT_URL --rev $CI_COMMIT_SHA path-to-file

[aguschin]

It turns out the same happens in GitHub Actions: https://github.com/iterative/yolov5/runs/2893101649?check_suite_focus=true
to follow the same approach here, we can use

dvc get https://github.com/$GITHUB_REPOSITORY --rev $GITHUB_SHA path-to-file

[jorgeorpinel]

the issue here is that the repo being cloned does have a HEAD ... but apparently refs/remotes/origin/HEAD isn't created when you clone a repo that is in a detached state

@pmrowla I see. Maybe addressing #3590 (comment) (applies to list/get/import) would also prevent this. Cc @aguschin so yes, get/import . wouldn't re-clone the repo in that case. I.e. this issue may be a dupe.

[jorgeorpinel]

Wait according to #3590 (comment) at least list already doesn't re-clone local repos. And then that issue veered into some other UI-related discussions. So it probably makes sense to keep this one about get/import specifically.

[pmrowla]

@jorgeorpinel The comments in #3590 are specific to list, that change does not apply to get.

This issue is marked P3 because we can potentially change get to not clone the local repo. However, that change will only
fix the issue for when the user is trying to get the HEAD revision.

The real problem here is that the user's local repo (.) will only have the specific revisions fetched into the runner env by gitlab/github CI, so using

dvc get . --rev <any other branch or rev>

won't work at all in these CI environments.

If the user wants consistent dvc get behavior in these CI envs, they should use the proper git remote URL (and not the local repo .) as noted above.

[jorgeorpinel]

local repo (.) will only have the specific revisions fetched into the runner env by gitlab/github CI

Yeah but as you suggest, that's not something DVC can do much about. So I basically see this issue as a request to not re-clone local repos with get/import.

[aguschin]

Not sure it should be reported here, but I encountered the issue with this approach while trying to do this in CI job triggered by Studio. In that job $GITHUB_SHA references to some unknown commit which could not be found anywhere, thus dvc get https://github.com/$GITHUB_REPOSITORY --rev $GITHUB_SHA path-to-file fails:

+ dvc get https://github.com/iterative/yolov5 --rev a3a326f7237b5e997fb69f3611f51f7a80fcfda5 yolov5s.pt
ERROR: failed to get 'yolov5s.pt' from 'https://github.com/iterative/yolov5' - unknown Git revision 'a3a326f7237b5e997fb69f3611f51f7a80fcfda5'

complete log here: https://github.com/iterative/yolov5/runs/2893514130?check_suite_focus=true

Not sure what is the right way to handle this in Github Actions, any suggestions would be appreciated. Right now I've hardcoded dvc get to download file from the default branch, but it doesn't seem like general solution.

[pmrowla]

One other question I have here: what's the reason for using dvc get . yolo5s.pt instead of dvc pull yolo5s.pt?

[aguschin]

@pmrowla, yolov5s.pt is not an output of dvc stage. It is versioned manually. Because of this, dvc pull is not working.

[pmrowla]

@aguschin it has a DVC file though? https://github.com/iterative/yolov5/blob/dvc/yolov5s.pt.dvc