git status with 8GB file VERY slow (over one minute)

[jhattongfs]

Describe the bug

Running git status is very slow if there is an 8GB file present in LFS. Files of different sizes, such as 9GB, work as expected.

Git is apparently reading back the entire 8GB file every time. This takes over 1 minute on my box.

git status never shows the file as modified. It just reads the file back every time and discovers nothing has changed.

This issue makes any command that shows file status very slow, including tab-complete and commit.

To Reproduce

git init
dd if=/dev/zero of=8gb bs=1024 count=$((1024*1024*8))
git lfs track 8gb
git add .
git commit
time git status
time git status
time git status

Expected behavior
Git status to take less than one second.

System environment

Windows with Git-2.36.0-64-bit.exe

The problem is present on old versions as well.

Output of git lfs env

git-lfs/3.1.4 (GitHub; windows amd64; go 1.17.8)
git version 2.36.0.windows.1

LocalWorkingDir=C:\Users\JHatton\Desktop\tmp\8g
LocalGitDir=C:\Users\JHatton\Desktop\tmp\8g.git
LocalGitStorageDir=C:\Users\JHatton\Desktop\tmp\8g.git
LocalMediaDir=C:\Users\JHatton\Desktop\tmp\8g.git\lfs\objects
LocalReferenceDirs=
TempDir=C:\Users\JHatton\Desktop\tmp\8g.git\lfs\tmp
ConcurrentTransfers=2
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=C:\Users\JHatton\Desktop\tmp\8g.git\lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context
Any other relevant context about the problem here.

[bk2204]

Hey,

This is definitely an issue in Git and should be reported to the Git for Windows issue tracker. Git LFS doesn't control whether git status should re-read the file; only Git determines that. It uses the data in the index to see if it thinks the data is stale and re-reads it if it is or if it can't be sure that it isn't.

My guess as to what's happening here is that this is a condition where the file is racily clean. That is, the file is actually clean, but the modification timestamp, for whatever reason, is actually the same as the index. In that case, we can't know whether the file is up to date, or if the index was refreshed immediately before the update to the file, and as such, Git needs to re-read the file.

Now, how that happens is unknown. It could be you have some sort of program scanning data, such as a third-party antivirus or monitoring program that does that, in which case you should absolutely remove it and use the default Windows antivirus. Third-party antivirus and firewall programs are known to break Git in a variety of ways.

It could also be that it's something else entirely and it's not a racily clean issue, but ultimately this problem, which we do see from time to time (and almost always on Windows) is an issue related to Git and is out of the control of Git LFS.

So I'm going to close this, but I encourage you to reach out to the Git for Windows project at https://github.com/git-for-windows/git. The maintainers of that project know me and will ping me on your issue if they need help.

[Arcitec]

Thanks, with your help I figured out out that my Powerline "git status" was hanging because git status was internally doing the Git-LFS Refresh index: 100% (302/302), done.... and I figured out that me changing the timestamps of all local files was the reason for the re-verification.

We have a script which corrects all local timestamps for all files from the repo (submodule) where we have Git-LFS files. It was a simple matter of putting a git -C "${THE_REPO_WITH_GIT_LFS_DATA}" status after we fix the timestamps, to force it to re-verify all local files. It's a very quick operation, running in maybe 20 seconds, but it never finished at all if running via a Powerline status, so it was important that we fixed this.

I wonder if there's a better way than doing git status. But I guess not. I guess that's where git decides to re-verify the files due to changed timestamps.