Cloning an LFS-enabled repository hangs forever on case-insensitive filesystems in some circumstances

[apetresc]

I have a sample repo that reproduces this bug here, with instructions. Cloning it on Windows or case-insensitive macOS HFS partitions fails.

Detailed steps to reproduce are in the README and commit history, but I'll paste them here:

1. Create a new repository without Git LFS
2. On a case-insensitive filesystem (such as Windows or a default HFS partition on macOS), commit two files (even in different directories!) whose filenames match, except for the case of the extension. For example, in this repository, there is img/a/1.jpg and img/b/1.JPG.
3. Enable Git LFS for the repository, and start tracking one or both of the extensions. In this repository, I added *.jpg filter=lfs diff=lfs merge=lfs -text to .gitattributes.
4. Push and then try to re-clone the repository on a case-insensitive filesystem. The clone will hang forever. For example, cloning this repository on Windows or macOS results in:

$ git clone git@github.com:apetresc/git-lfs-bug
Cloning into 'git-lfs-bug'...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 16 (delta 0), reused 16 (delta 0), pack-reused 0
Receiving objects: 100% (16/16), 383.15 KiB | 0 bytes/s, done.
Downloading img/a/1.jpg (33.38 KB)

followed by a permanent hang that must be ended with SIGTERM. However, cloning on a case-sensitive filesystem like Linux will work just fine!

Obviously this is a breaking bug that acts as a dealbreaker for using Git LFS if Windows/macOS needs to be supported. I haven't gotten as far as a diagnosis yet, I am hoping someone on the Git LFS team will be able to figure it out on sight without me having to dig into LFS internals :)

[apetresc]

UPDATE: This problem does not occur if I use git lfs clone instead of git clone. So maybe that means the problem is in the smudge filter? Not really certain, but this is certainly a very good workaround. Probably still worth investigating the core issue though.

[ttaylorr]

Probably still worth investigating the core issue though.

I agree, and sorry that you're encountering this bug. I'll put this on my list of things to take a look at. I think a good first step would be getting this reproducible in a PR with a failing test, so I'll start there.

[neslekkim]

I think I got the same problem now, first I got an problem that we didn't have git-lfs installed on our buildserver (teamcity), installed it into the same folder as git.exe.
But everytime we tried to build, we got this error:

git-lfs filter-process: git-lfs: command not found
fatal: The remote end hung up unexpectedly
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

When running taskmanager I found that git.exe is run for multiple folders??, so copying git-lfs.exe into .\mingw64\libexec\git-core and .\mingw64\bit helped, but now it hangs forever when downloading the repository, so our buildserver is rendered useless, and we cannot make new clones of our repository..
and the hang is the same as above..
(we use Bitbucket cloud)

Is there any workaround for this?

[apetresc]

@neslekkim I'm not sure if your hang is the same underlying issue as this one, but there is a very simple workaround for mine: instead of using git clone directly, use git-lfs clone. If that doesn't help, it's likely a different issue than mine, and you should file a separate ticket to get to the bottom of it. Make sure to include the output when you run with GIT_TRACE=1!

[neslekkim]

Where do I put the git_trace=1 ? environmentvariable?

[neslekkim]

Yes, using git-lfs clone helped getting the files again, except that the lfs files was not cloned, only files containing stuff like this:
version https://git-lfs.github.com/spec/v1
oid sha256:111e2a1e23914544dffc77f3744e6ce5ab11aa60bf3098df33bd62ff2e0b8ce7
size 13490919

trying git-lfs fetch after this, then I get this:
Fetching master
Git LFS: (0 of 1 files) 0 B / 12.87 MB
LFS: Git credentials for https://A...en@bitbucket.org/sdfdsf/project.git not found.
Warning: errors occurred

[neslekkim]

If I do "git config credential.helper store" then "git pull" so I'm asked for the password again, then "git-lfs fetch" suddenly work..
Is the problem related to this?, not case insensitive filesystem?
Not sure how I can teach teamcity to do all this mumbo to get things working again..

[ttaylorr]

Is the problem related to this?, not case insensitive filesystem?

Judging from the conversation, it seems like this is likely related to your credential helper.

[evitolins]

We are having some issues that seem to spawn from a case-sensitive filenames situation.

In our repo, I've found several files which were committed recently with a UPPER-CASE extension. The lower-case equivelent is definitely within our .gitattributes, but seems the rules did not catch their upper-case counter-parts during commit/push.

When an end-user clones/pulls/checkout... these upper-cased files are expecting LFS pointer files, but fail as they were not tracked correctly when committed/pushed.

On Windows 10 (Git Bash and SourceTree)

The error when checking out this commit, the issue seems to hang, with no messages printed.

On MacOS 10.11.6 (CLI)

The errors reported and printed as such, but continue until completion as it should.

Pointer file error: Unable to parse pointer at: "Assets/Origin/Models/Barrels.FBX"
Pointer file error: Unable to parse pointer at: "Assets/Origin/Models/Boat.FBX"
Pointer file error: Unable to parse pointer at: "Assets/Origin/Models/Boat_Mesh.FBX"
Pointer file error: Unable to parse pointer at: "Assets/Origin/Models/Chest.FBX"
...

I ran a search within the working tree, and ALL 92 files in the with upper-case file extensions had matching pointer error reported.

Bonus

# Find all files with uppercase extensions
find -E . -regex '.*\.[A-Z]{1,4}' | wc -l

[dipyalov]

@evitolins Did you manage to find a workaround? Cause upper-case extensions are a real PITA in gamedev scenarios. And some artist tools, like Photoshop have a tendency to make upper-case extensions.

[evitolins]

@dipyalov no, we never did find a solution directly. We've yet to move to LFS 2.0 and maybe it has been resolved. For now, we've been relying on pre-commit hooks to avoid any incidents. If the team decides to open-source it, i'll be sure to share it here. ;)

[dipyalov]

@evitolins I've had the same problem in LFS 2.0
It seems, like I've managed to make my repo working by mentioning all the uppercase file extensions in .gitattributes

[evitolins]

@dipyalov We'd also considered that, but it felt dirty and was sad to go that route 😭 😆 . With that said, it's probably WAY easier than having to deal with hooks ;). Well done.

[evitolins]

Actually @dipyalov... there WAS a reason we decided NOT just adding tracking for uppercase extensions. Cases like "Png", "PnG", etc... would not be detected. It would definitely be annoying and cumbersome to cover all permutations of the upper/lower case possibilities within the .gitattributes file.

[junebug12851]

I'm having a similar issue and running latest git-lfs - trying git lfs clone now, it printed deprecated warning message but seems much more responsive.

Before with git clone

Cloning into '###'...
remote: 
remote:                    vSTs   
remote:                  vSTSVSTSv
remote:                vSTSVSTSVST
remote: VSTS         vSTSVSTSVSTSV
remote: VSTSVS     vSTSVSTSV STSVS
remote: VSTSVSTSvsTSVSTSVS   TSVST
remote: VS  tSVSTSVSTSv      STSVS
remote: VS   tSVSTSVST       SVSTS
remote: VS tSVSTSVSTSVSts    VSTSV
remote: VSTSVST    SVSTSVSTs VSTSV
remote: VSTSv        STSVSTSVSTSVS
remote:                VSTSVSTSVST
remote:                  VSTSVSTs
remote:                    VSTs    (TM)
remote: 
remote:  Microsoft (R) Visual Studio (R) Team Services
remote: 
remote: Found 2347 objects to send. (67 ms)
Receiving objects: 100% (2347/2347), 1.65 MiB | 861.00 KiB/s, done.
Resolving deltas: 100% (786/786), done.
<Said something about filtering and got hung up mid-way through for an hour before I force closed it>

With git lfs clone

WARNING: 'git lfs clone' is deprecated and will not be updated
          with new flags from 'git clone'

'git clone' has been updated in upstream Git to have comparable
speeds to 'git lfs clone'.
Cloning into '###'...
remote: 
remote:                    vSTs   
remote:                  vSTSVSTSv
remote:                vSTSVSTSVST
remote: VSTS         vSTSVSTSVSTSV
remote: VSTSVS     vSTSVSTSV STSVS
remote: VSTSVSTSvsTSVSTSVS   TSVST
remote: VS  tSVSTSVSTSv      STSVS
remote: VS   tSVSTSVST       SVSTS
remote: VS tSVSTSVSTSVSts    VSTSV
remote: VSTSVST    SVSTSVSTs VSTSV
remote: VSTSv        STSVSTSVSTSVS
remote:                VSTSVSTSVST
remote:                  VSTSVSTs
remote:                    VSTs    (TM)
remote: 
remote:  Microsoft (R) Visual Studio (R) Team Services
remote: 
remote: Found 2347 objects to send. (300 ms)
Receiving objects: 100% (2347/2347), 1.65 MiB | 495.00 KiB/s, done.
Resolving deltas: 100% (786/786), done.
Git LFS: (200 of 293 files) 262.22 MB / 853.25 MB  

It's still downloading so I'll wait but far more responsive and fast. I'll edit this message below when it's done on the results.

EDIT

Receiving objects: 100% (2347/2347), 1.65 MiB | 495.00 KiB/s, done.
Resolving deltas: 100% (786/786), done.
Git LFS: (293 of 293 files) 853.25 MB / 853.25 MB

It completed just fine without a hiccup, the work-around works. So glad I found this thread, however this bug is really old, is it difficult to fix?

For reference I'm on a Case-Insensitive Mac OSX. I would have chosen a case-sensitive when I last formatted it but then everything else doesn't work, from programming tools like IntelliJ to games - they all get hung up and refuse to work if it's case sensitive but git-lfs apparently doesn't work if it's case-insensitive. I just can't make everyone happy lol. But glad I found this work-around.

[ttaylorr]

@apetresc I think this may be a different bug. If you are still interested in a fix, can you please open a new issue? Using git clone and git lfs clone perform the checkout procedure in two different ways, so I'm wondering if there's a discrepancy between the two causing one to hang and the other to not.

If you're interested in opening a new issue, please run GIT_TRACE_PACKET=1 git clone path/to/repo.git and post the output.

EDIT: I'm going to close this issue, since there are now a number of potentially unrelated issues all in the same place. In an effort to push the conversation towards a few more well-defined issue, I'm going to close this one for now.