Clone fails if source has LFS files

[minrk]

Due to git-lfs/git-lfs#1207 , cloning from a local repo fails if it contains files stored in git-lfs. Since conda-build seems to always clone to a cache first, then clone from there for the build (logical), repos storing files in LFS fail with errors like:

Cloning into '/Users/benjaminrk/conda/conda-bld/fenics_1476434449181/work'...
done.
Downloading demo/documented/bcs/aneurysm.xml.gz (686.73 KB)
Error downloading object: demo/documented/bcs/aneurysm.xml.gz (95ece98fe8dbf0e3ecc46cdf01cf0a9cbd65da1f1ebc4a27eebc564dad6f978c)

Errors logged to /Users/benjaminrk/conda/conda-bld/fenics_1476434449181/work/.git/lfs/objects/logs/20161014T104139.022737313.log

where the error in the log has a key phrase, such as unsupported protocol scheme "".

If there were a way to mark these sources as not-cacheable (or if anyone has an idea for how to make the lfs fetches work), that would help me.

[msarahan]

Sorry, I'm totally new to LFS. I'm open to PRs on this, but don't have time to investigate further right now. It seems like there could/should be a way to tell conda-build to bypass the cache as the fastest possible solution here.

[minrk]

Allowing packages to disable the cache seems like a totally adequate solution for now. I'll look into a PR.

[adamlabadorf]

Sorry to pick back up on this old thread, but I'm running into this issue and based on the threads I've seen I'm not sure how to get around it until the proposed fixes are integrated. Is there an accepted workaround for this problem? I'm trying to build a package that has some LFS files tracked and the only way I've been able to do it is to branch and strip it all out when I need to make a new release, which is very not fun.

Cheers.

[lkilcher]

I also ran into this problem. A workaround is to use a pip install as the source repo. In order for this to work, you do have to create the pypi package first. This worked for me though!

[farleylai]

There should be an option somewhere in meta.yaml or so.
Otherwise, export GIT_LFS_SKIP_SMUDGE=1 should have LFS files to be skipped for conda-build to clone a git repo.
However, if checking out LFS files is necessary e.g. test assets, this issue must be fixed.

[ernstluring]

I also encountered this issue, so I decided to dive into this and I found a solution!

The problem consists of two parts:

1. git lfs only supports cloning from a file URL from version 2.10.0 or higher (see github.com LFS error: 'batch requested: missing protocol' git-lfs/git-lfs#3041). And conda-build uses this when cloning from the mirror directory in the cache to the destination directory. The problem here is that when you update git, git lfs is not automatically updated to its newest version. So you have to do this manually. For that, check this installation guide: https://github.com/git-lfs/git-lfs/wiki/Installation.
2. When conda-build creates the mirror directory in the cache, it uses git clone --mirror but this does not get the lfs directory too. For this, I made a PR here: Fixes the git lfs error by adding git lfs fetching #4318

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[h-vetinari]

I still see this error with conda-build 3.24.0 in conda-forge/staged-recipes#17082 for trying to pull in a submodule of torchaudio...

  - git_url: https://github.com/flashlight/text.git
    git_rev: 98028c7da83d66c2aba6f5f8708c063d266ca5a4
    folder: third_party/flashlight-text

which fails with:

Cloning into bare repository '/home/conda/staged-recipes/build_artifacts/git_cache/github.com/flashlight/text.git'...
git: 'lfs' is not a git command. See 'git --help'.

Is it possible that #4318 did not enable the git_url case fully and/or correctly?