git status extremely slow if any 8GBi file is in the repo

[jhattongfs]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

64 bit

$ git --version --build-options

git version 2.36.0.windows.1
cpu: x86_64
built from commit: ea1e13f73339d57cbe81a0bae6fba669aaccf656
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows 10, 64 bit. 21H1

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19043.1645]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

nope

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

git bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git init
dd if=/dev/zero of=8gb bs=1024 count=$((1024*1024*8))
git lfs track 8gb
git add .
git commit
time git status
time git status
time git status

• What did you expect to occur after running these commands?

I expected the time to be less than one second.

• What actually happened instead?

git commit and all the git status commands take over one minute.

Files that are larger or smaller than 8GB seem to work as expected.

@bk2204 says this is not related to LFS. See 4983.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

N/A

[PhilipOakley]

Files that are larger or smaller than 8GB seem to work as expected.

Are you stating that the file length must be exactly 8GBi or 2^33, not one byte more or less? Just wondering if/where the problem roll-over happens.

[jhattongfs]

I didn't try it to one byte, but the the only files I have trouble with are exactly 2^33. I found it when adding a multi-part zip file with 8GBi file chunks. (I didn't try other multiples of 4GBi either).

[jhattongfs]

2^33 - 1 bytes works without any issues.

[PhilipOakley]

Did you try 2^33+1 as well?

Just wondering if git's squeezing the size value into a 32bit to get funny roll over issues or its the racy problem brian mentioned.

[jhattongfs]

I have re-confirmed that 2^33 plus or minus 1 byte works. 2^33 does not.

The truncate command is helpful for testing this.

[florcajal]

hola, necesito de su ayuda, no puedo instalar GIT. cuando termina la instalacion me aparece un cuadro donde dice c:program filesGIT/git-bash.exe
create process failed; code2

[dscho]

@florcajal please refrain from adding content to this ticket that is unrelated to the original report. Thank you.

@jhattongfs could you re-run with GIT_TRACE_PERF set and analyze the output?

[jhattongfs]

I think I got this right. The results are below.

git status summary

read-cache.c:1728	data		78.63	78.51	index	....refresh/sum_lstat:2
read-cache.c:1729	data		78.63	78.51	index	....refresh/sum_scan:1
read-cache.c:1730	region_leave		78.63	78.51	index	..label:refresh
progress.c:339	data	r1	78.63	78.51	progress	..total_objects:2
progress.c:346	region_leave	r1	78.63	78.51	progress	label:Refresh index
wt-status.c:818	region_enter	r1	78.64		status	label:worktrees
diffcore-rename.c:1401	region_enter	r1	78.64		diff	..label:setup
diffcore-rename.c:1454	region_leave	r1	78.64	0.00	diff	..label:setup
diffcore-rename.c:1633	region_enter	r1	78.64		diff	..label:write back to queue
diffcore-rename.c:1710	region_leave	r1	78.64	0.00	diff	..label:write back to queue
wt-status.c:820	region_leave	r1	78.64	0.00	status	label:worktrees
wt-status.c:827	region_enter	r1	78.64		status	label:index
unpack-trees.c:1789	region_enter	r1	78.65		unpack_trees	..label:unpack_trees
unpack-trees.c:1884	region_enter	r1	78.65		unpack_trees	....label:traverse_trees
unpack-trees.c:1886	region_leave	r1	78.65	0.00	unpack_trees	....label:traverse_trees
unpack-trees.c:1976	region_leave	r1	78.65	0.00	unpack_trees	..label:unpack_trees
diffcore-rename.c:1401	region_enter	r1	78.65		diff	..label:setup
diffcore-rename.c:1454	region_leave	r1	78.65	0.00	diff	..label:setup
diffcore-rename.c:1633	region_enter	r1	78.65		diff	..label:write back to queue
diffcore-rename.c:1710	region_leave	r1	78.65	0.00	diff	..label:write back to queue
wt-status.c:829	region_leave	r1	78.65	0.00	status	label:index
wt-status.c:832	region_enter	r1	78.65		status	label:untracked
dir.c:2958	region_enter	r1	78.65		dir	..label:read_directory
name-hash.c:585	region_enter	r1	78.65		index	....label:name-hash-init
name-hash.c:606	region_leave	r1	78.65	0.00	index	....label:name-hash-init
dir.c:2926	data	r1	78.65	0.00	read_directo	....path:
dir.c:2934	data	r1	78.65	0.00	read_directo	....directories-visited:1
dir.c:2936	data	r1	78.65	0.00	read_directo	....paths-visited:5
dir.c:2981	region_leave	r1	78.65	0.00	dir	..label:read_directory
wt-status.c:834	region_leave	r1	78.65	0.00	status	label:untracked
read-cache.c:3178	region_enter	r1	78.65		index	label:do_write_index C:/Users/JHatton/Desktop/tmp/8GBi/.git/index.lock
cache-tree.c:532	region_enter	r1	78.65		cache_tree	..label:write
cache-tree.c:534	region_leave	r1	78.65	0.00	cache_tree	..label:write
read-cache.c:3140	data	r1	78.65	0.00	index	..write/version:2
read-cache.c:3142	data	r1	78.65	0.00	index	..write/cache_nr:2
read-cache.c:3181	region_leave	r1	78.65	0.00	index	label:do_write_index C:/Users/JHatton/Desktop/tmp/8GBi/.git/index.lock
wt-status.c:2510	data	r1	78.66	78.66	status	count/changed:0
wt-status.c:2511	data	r1	78.66	78.66	status	count/untracked:2
wt-status.c:2513	data	r1	78.66	78.66	status	count/ignored:0
wt-status.c:2515	region_enter	r1	78.66		status	label:print
wt-status.c:2536	region_leave	r1	78.66	0.00	status	label:print
wrapper.c:621	data	r1	78.66	78.66	fsync	fsync/writeout-only:0
wrapper.c:622	data	r1	78.66	78.66	fsync	fsync/hardware-flush:0
..._win32_process_info.c:166	data_json	r1	78.66	78.66	process	windows/memory:{"PageFaultCount":5176,"PeakWorkingSetSize":10301440,"PeakPagefileUsage":8015872}
git.c:719	exit		78.66			code:0
tree-walk.c:185	data_json	r1	78.66	78.66	traverse_tre	statistics:{"traverse_trees_count":1,"traverse_trees_max_depth":1}
run-command.c:977	child_exit		78.69	78.57		[ch0] pid:11500 code:0
trace2/tr2_tgt_perf.c:215	atexit		78.69			code:0

Full log from git status

08:33:46.960466 common-main.c:49	d0	main	version					2.36.0.windows.1
08:33:46.964432 common-main.c:50	d0	main	start		0.083613			'C:\Program Files\Git\mingw64\bin\git.exe' rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree --short HEAD
08:33:46.981626 ..._win32_process_info.c:118	d0	main	data_json	r0	0.100825	0.100825	process	windows/ancestry:["bash.exe","bash.exe","bash.exe","bash.exe"]
08:33:46.982288 git.c:460	d0	main	cmd_name					rev-parse (rev-parse)
08:33:46.990224 repository.c:136	d0	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:46.993531 wrapper.c:621	d0	main	data	r1	0.112494	0.112494	fsync	fsync/writeout-only:0
08:33:46.993531 wrapper.c:622	d0	main	data	r1	0.112593	0.112593	fsync	fsync/hardware-flush:0
08:33:46.994192 ..._win32_process_info.c:166	d0	main	data_json	r1	0.113136	0.113136	process	windows/memory:{"PageFaultCount":5813,"PeakWorkingSetSize":8835072,"PeakPagefileUsage":5947392}
08:33:46.994192 git.c:719	d0	main	exit		0.113211			code:0
08:33:46.994192 trace2/tr2_tgt_perf.c:215	d0	main	atexit		0.113276			code:0
08:33:47.324622 common-main.c:49	d0	main	version					2.36.0.windows.1
08:33:47.333219 common-main.c:50	d0	main	start		0.092072			'C:\Program Files\Git\mingw64\bin\git.exe' config --bool core.sparseCheckout
08:33:47.358904 ..._win32_process_info.c:118	d0	main	data_json	r0	0.117395	0.117395	process	windows/ancestry:["bash.exe","bash.exe","bash.exe","bash.exe"]
08:33:47.367501 repository.c:136	d0	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:47.367501 git.c:460	d0	main	cmd_name					config (config)
08:33:47.368824 wrapper.c:621	d0	main	data	r1	0.127759	0.127759	fsync	fsync/writeout-only:0
08:33:47.368824 wrapper.c:622	d0	main	data	r1	0.127858	0.127858	fsync	fsync/hardware-flush:0
08:33:47.370148 ..._win32_process_info.c:166	d0	main	data_json	r1	0.128572	0.128572	process	windows/memory:{"PageFaultCount":5756,"PeakWorkingSetSize":8806400,"PeakPagefileUsage":4161536}
08:33:47.370148 git.c:719	d0	main	exit		0.128682			code:1
08:33:47.370148 trace2/tr2_tgt_perf.c:215	d0	main	atexit		0.128771			code:1
08:33:52.340687 common-main.c:49	d0	main	version					2.36.0.windows.1
08:33:52.348475 common-main.c:50	d0	main	start		0.083014			'C:\Program Files\Git\mingw64\bin\git.exe' status
08:33:52.364347 ..._win32_process_info.c:118	d0	main	data_json	r0	0.099060	0.099060	process	windows/ancestry:["bash.exe","bash.exe"]
08:33:52.374930 repository.c:136	d0	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:52.376254 git.c:460	d0	main	cmd_name					status (status)
08:33:52.378896 read-cache.c:2446	d0	main	region_enter	r1	0.113330		index	label:do_read_index .git/index
08:33:52.380880 cache-tree.c:628	d0	main	region_enter	r1	0.115027		cache_tree	..label:read
08:33:52.380880 cache-tree.c:630	d0	main	region_leave	r1	0.115146	0.000119	cache_tree	..label:read
08:33:52.380880 read-cache.c:2392	d0	main	data	r1	0.115257	0.001927	index	..read/version:2
08:33:52.380880 read-cache.c:2394	d0	main	data	r1	0.115352	0.002022	index	..read/cache_nr:2
08:33:52.380880 read-cache.c:2451	d0	main	region_leave	r1	0.115457	0.002127	index	label:do_read_index .git/index
08:33:52.381541 progress.c:268	d0	main	region_enter	r1	0.116137		progress	label:Refresh index
08:33:52.381541 read-cache.c:1646	d0	main	region_enter		0.116253		index	..label:refresh
08:33:52.383526 run-command.c:721	d0	main	child_start		0.117909			....[ch0] class:subprocess argv:['git-lfs filter-process']
08:33:52.867788 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:52.874383 common-main.c:50	d1	main	start		0.076544			git version
08:33:52.894867 ..._win32_process_info.c:118	d1	main	data_json	r0	0.096527	0.096527	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:52.894867 git.c:460	d1	main	cmd_name					version (status/version)
08:33:52.894867 wrapper.c:621	d1	main	data	r0	0.097008	0.097008	fsync	fsync/writeout-only:0
08:33:52.894867 wrapper.c:622	d1	main	data	r0	0.097071	0.097071	fsync	fsync/hardware-flush:0
08:33:52.895525 ..._win32_process_info.c:166	d1	main	data_json	r0	0.097647	0.097647	process	windows/memory:{"PageFaultCount":6303,"PeakWorkingSetSize":8802304,"PeakPagefileUsage":4173824}
08:33:52.895525 git.c:719	d1	main	exit		0.097724			code:0
08:33:52.895525 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.097788			code:0
08:33:53.071530 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:53.078175 common-main.c:50	d1	main	start		0.077276			git -c filter.lfs.smudge= -c filter.lfs.clean= -c filter.lfs.process= -c filter.lfs.required=false rev-parse --git-dir --show-toplevel
08:33:53.097354 ..._win32_process_info.c:118	d1	main	data_json	r0	0.096762	0.096762	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:53.098015 git.c:460	d1	main	cmd_name					rev-parse (status/rev-parse)
08:33:53.105951 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:53.107274 wrapper.c:621	d1	main	data	r1	0.106511	0.106511	fsync	fsync/writeout-only:0
08:33:53.107274 wrapper.c:622	d1	main	data	r1	0.106586	0.106586	fsync	fsync/hardware-flush:0
08:33:53.107935 ..._win32_process_info.c:166	d1	main	data_json	r1	0.107133	0.107133	process	windows/memory:{"PageFaultCount":6358,"PeakWorkingSetSize":8814592,"PeakPagefileUsage":4182016}
08:33:53.107935 git.c:719	d1	main	exit		0.107219			code:0
08:33:53.107935 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.107287			code:0
08:33:53.830355 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:53.838658 common-main.c:50	d1	main	start		0.088459			git rev-parse --is-bare-repository
08:33:53.863722 ..._win32_process_info.c:118	d1	main	data_json	r0	0.113846	0.113846	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:53.864384 git.c:460	d1	main	cmd_name					rev-parse (status/rev-parse)
08:33:53.871569 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:53.872713 wrapper.c:621	d1	main	data	r1	0.122702	0.122702	fsync	fsync/writeout-only:0
08:33:53.872713 wrapper.c:622	d1	main	data	r1	0.122784	0.122784	fsync	fsync/hardware-flush:0
08:33:53.873341 ..._win32_process_info.c:166	d1	main	data_json	r1	0.123454	0.123454	process	windows/memory:{"PageFaultCount":6351,"PeakWorkingSetSize":8802304,"PeakPagefileUsage":4177920}
08:33:53.874056 git.c:719	d1	main	exit		0.123601			code:0
08:33:53.874056 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.123694			code:0
08:33:54.045380 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:54.053916 common-main.c:50	d1	main	start		0.085934			git config --includes --local lfs.repositoryformatversion
08:33:54.080957 ..._win32_process_info.c:118	d1	main	data_json	r0	0.113318	0.113318	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:54.089601 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:54.090267 git.c:460	d1	main	cmd_name					config (status/config)
08:33:54.090267 wrapper.c:621	d1	main	data	r1	0.122576	0.122576	fsync	fsync/writeout-only:0
08:33:54.090267 wrapper.c:622	d1	main	data	r1	0.122672	0.122672	fsync	fsync/hardware-flush:0
08:33:54.090964 ..._win32_process_info.c:166	d1	main	data_json	r1	0.123236	0.123236	process	windows/memory:{"PageFaultCount":6344,"PeakWorkingSetSize":8802304,"PeakPagefileUsage":4165632}
08:33:54.091585 git.c:719	d1	main	exit		0.123469			code:0
08:33:54.091585 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.123563			code:0
08:33:54.385753 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:54.393409 common-main.c:50	d1	main	start		0.082031			git config --includes -l
08:33:54.419734 ..._win32_process_info.c:118	d1	main	data_json	r0	0.108135	0.108135	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:54.429619 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:54.430270 git.c:460	d1	main	cmd_name					config (status/config)
08:33:54.431594 wrapper.c:621	d1	main	data	r1	0.120262	0.120262	fsync	fsync/writeout-only:0
08:33:54.432254 wrapper.c:622	d1	main	data	r1	0.120337	0.120337	fsync	fsync/hardware-flush:0
08:33:54.432254 ..._win32_process_info.c:166	d1	main	data_json	r1	0.120904	0.120904	process	windows/memory:{"PageFaultCount":6351,"PeakWorkingSetSize":8798208,"PeakPagefileUsage":4165632}
08:33:54.432916 git.c:719	d1	main	exit		0.121013			code:0
08:33:54.432916 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.121104			code:0
08:33:54.608513 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:54.615789 common-main.c:50	d1	main	start		0.090036			git rev-parse --is-bare-repository
08:33:54.635627 ..._win32_process_info.c:118	d1	main	data_json	r0	0.109730	0.109730	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:54.635627 git.c:460	d1	main	cmd_name					rev-parse (status/rev-parse)
08:33:54.644225 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:54.645619 wrapper.c:621	d1	main	data	r1	0.119722	0.119722	fsync	fsync/writeout-only:0
08:33:54.645619 wrapper.c:622	d1	main	data	r1	0.119826	0.119826	fsync	fsync/hardware-flush:0
08:33:54.646210 ..._win32_process_info.c:166	d1	main	data_json	r1	0.120425	0.120425	process	windows/memory:{"PageFaultCount":6353,"PeakWorkingSetSize":8806400,"PeakPagefileUsage":4182016}
08:33:54.646210 git.c:719	d1	main	exit		0.120538			code:0
08:33:54.646210 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.120629			code:0
08:33:54.796899 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:54.805382 common-main.c:50	d1	main	start		0.079034			git config --includes -l --blob :.lfsconfig
08:33:54.834348 ..._win32_process_info.c:118	d1	main	data_json	r0	0.108080	0.108080	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:54.844930 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:54.844930 git.c:460	d1	main	cmd_name					config (status/config)
08:33:54.846267 read-cache.c:2446	d1	main	region_enter	r1	0.119497		index	label:do_read_index .git/index
08:33:54.848240 cache-tree.c:628	d1	main	region_enter	r1	0.121627		cache_tree	..label:read
08:33:54.848240 cache-tree.c:630	d1	main	region_leave	r1	0.121707	0.000080	cache_tree	..label:read
08:33:54.848240 read-cache.c:2392	d1	main	data	r1	0.121779	0.002282	index	..read/version:2
08:33:54.848240 read-cache.c:2394	d1	main	data	r1	0.121840	0.002343	index	..read/cache_nr:2
08:33:54.848240 read-cache.c:2451	d1	main	region_leave	r1	0.121904	0.002407	index	label:do_read_index .git/index
08:33:54.848903 usage.c:79	d1	main	error					unable to resolve config blob ':.lfsconfig'
08:33:54.849050 usage.c:60	d1	main	error					error processing config file(s)
08:33:54.849050 wrapper.c:621	d1	main	data	r1	0.122310	0.122310	fsync	fsync/writeout-only:0
08:33:54.849050 wrapper.c:622	d1	main	data	r1	0.122381	0.122381	fsync	fsync/hardware-flush:0
08:33:54.849578 ..._win32_process_info.c:166	d1	main	data_json	r1	0.122908	0.122908	process	windows/memory:{"PageFaultCount":6395,"PeakWorkingSetSize":8802304,"PeakPagefileUsage":4165632}
08:33:54.849578 usage.c:74	d1	main	exit		0.122981			code:128
08:33:54.849578 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.123037			code:128
08:33:55.014421 common-main.c:49	d1	main	version					2.36.0.windows.1
08:33:55.021037 common-main.c:50	d1	main	start		0.080748			git config --includes -l --blob HEAD:.lfsconfig
08:33:55.046825 ..._win32_process_info.c:118	d1	main	data_json	r0	0.106209	0.106209	process	windows/ancestry:["git-lfs.exe","sh.exe","git.exe","bash.exe","bash.exe"]
08:33:55.056746 repository.c:136	d1	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:33:55.057407 git.c:460	d1	main	cmd_name					config (status/config)
08:33:55.066004 usage.c:79	d1	main	error					unable to resolve config blob 'HEAD:.lfsconfig'
08:33:55.066004 usage.c:60	d1	main	error					error processing config file(s)
08:33:55.066528 wrapper.c:621	d1	main	data	r1	0.126065	0.126065	fsync	fsync/writeout-only:0
08:33:55.066528 wrapper.c:622	d1	main	data	r1	0.126188	0.126188	fsync	fsync/hardware-flush:0
08:33:55.067567 ..._win32_process_info.c:166	d1	main	data_json	r1	0.126926	0.126926	process	windows/memory:{"PageFaultCount":6423,"PeakWorkingSetSize":8867840,"PeakPagefileUsage":4182016}
08:33:55.067567 usage.c:74	d1	main	exit		0.127041			code:128
08:33:55.067567 trace2/tr2_tgt_perf.c:215	d1	main	atexit		0.127122			code:128
08:35:10.896345 read-cache.c:1728	d0	main	data		78.630573	78.514320	index	....refresh/sum_lstat:2
08:35:10.899160 read-cache.c:1729	d0	main	data		78.633832	78.517579	index	....refresh/sum_scan:1
08:35:10.899160 read-cache.c:1730	d0	main	region_leave		78.633895	78.517642	index	..label:refresh
08:35:10.899160 progress.c:339	d0	main	data	r1	78.634016	78.517879	progress	..total_objects:2
08:35:10.899880 progress.c:346	d0	main	region_leave	r1	78.634143	78.518006	progress	label:Refresh index
08:35:10.907097 wt-status.c:818	d0	main	region_enter	r1	78.641588		status	label:worktrees
08:35:10.909052 diffcore-rename.c:1401	d0	main	region_enter	r1	78.643577		diff	..label:setup
08:35:10.909052 diffcore-rename.c:1454	d0	main	region_leave	r1	78.643843	0.000266	diff	..label:setup
08:35:10.909052 diffcore-rename.c:1633	d0	main	region_enter	r1	78.643903		diff	..label:write back to queue
08:35:10.909713 diffcore-rename.c:1710	d0	main	region_leave	r1	78.643977	0.000074	diff	..label:write back to queue
08:35:10.909783 wt-status.c:820	d0	main	region_leave	r1	78.644041	0.002453	status	label:worktrees
08:35:10.909783 wt-status.c:827	d0	main	region_enter	r1	78.644110		status	label:index
08:35:10.917083 unpack-trees.c:1789	d0	main	region_enter	r1	78.651295		unpack_trees	..label:unpack_trees
08:35:10.917712 unpack-trees.c:1884	d0	main	region_enter	r1	78.652232		unpack_trees	....label:traverse_trees
08:35:10.917712 unpack-trees.c:1886	d0	main	region_leave	r1	78.652314	0.000082	unpack_trees	....label:traverse_trees
08:35:10.918450 unpack-trees.c:1976	d0	main	region_leave	r1	78.652761	0.001466	unpack_trees	..label:unpack_trees
08:35:10.918450 diffcore-rename.c:1401	d0	main	region_enter	r1	78.652828		diff	..label:setup
08:35:10.918450 diffcore-rename.c:1454	d0	main	region_leave	r1	78.652883	0.000055	diff	..label:setup
08:35:10.918450 diffcore-rename.c:1633	d0	main	region_enter	r1	78.652938		diff	..label:write back to queue
08:35:10.918450 diffcore-rename.c:1710	d0	main	region_leave	r1	78.652991	0.000053	diff	..label:write back to queue
08:35:10.918450 wt-status.c:829	d0	main	region_leave	r1	78.653132	0.009022	status	label:index
08:35:10.918450 wt-status.c:832	d0	main	region_enter	r1	78.653193		status	label:untracked
08:35:10.919064 dir.c:2958	d0	main	region_enter	r1	78.653550		dir	..label:read_directory
08:35:10.919064 name-hash.c:585	d0	main	region_enter	r1	78.653617		index	....label:name-hash-init
08:35:10.919064 name-hash.c:606	d0	main	region_leave	r1	78.653682	0.000065	index	....label:name-hash-init
08:35:10.919064 dir.c:2926	d0	main	data	r1	78.653745	0.000195	read_directo	....path:
08:35:10.919064 dir.c:2934	d0	main	data	r1	78.653803	0.000253	read_directo	....directories-visited:1
08:35:10.919064 dir.c:2936	d0	main	data	r1	78.653857	0.000307	read_directo	....paths-visited:5
08:35:10.919670 dir.c:2981	d0	main	region_leave	r1	78.653912	0.000362	dir	..label:read_directory
08:35:10.919756 wt-status.c:834	d0	main	region_leave	r1	78.654017	0.000824	status	label:untracked
08:35:10.920522 read-cache.c:3178	d0	main	region_enter	r1	78.654793		index	label:do_write_index C:/Users/JHatton/Desktop/tmp/8GBi/.git/index.lock
08:35:10.920522 cache-tree.c:532	d0	main	region_enter	r1	78.654949		cache_tree	..label:write
08:35:10.920522 cache-tree.c:534	d0	main	region_leave	r1	78.655034	0.000085	cache_tree	..label:write
08:35:10.921186 read-cache.c:3140	d0	main	data	r1	78.656055	0.001262	index	..write/version:2
08:35:10.921854 read-cache.c:3142	d0	main	data	r1	78.656179	0.001386	index	..write/cache_nr:2
08:35:10.921996 read-cache.c:3181	d0	main	region_leave	r1	78.656289	0.001496	index	label:do_write_index C:/Users/JHatton/Desktop/tmp/8GBi/.git/index.lock
08:35:10.929655 wt-status.c:2510	d0	main	data	r1	78.664057	78.664057	status	count/changed:0
08:35:10.929655 wt-status.c:2511	d0	main	data	r1	78.664144	78.664144	status	count/untracked:2
08:35:10.929655 wt-status.c:2513	d0	main	data	r1	78.664206	78.664206	status	count/ignored:0
08:35:10.929655 wt-status.c:2515	d0	main	region_enter	r1	78.664262		status	label:print
08:35:10.931397 wt-status.c:2536	d0	main	region_leave	r1	78.665704	0.001442	status	label:print
08:35:10.931397 wrapper.c:621	d0	main	data	r1	78.665875	78.665875	fsync	fsync/writeout-only:0
08:35:10.931397 wrapper.c:622	d0	main	data	r1	78.665935	78.665935	fsync	fsync/hardware-flush:0
08:35:10.932439 ..._win32_process_info.c:166	d0	main	data_json	r1	78.666887	78.666887	process	windows/memory:{"PageFaultCount":5176,"PeakWorkingSetSize":10301440,"PeakPagefileUsage":8015872}
08:35:10.932674 git.c:719	d0	main	exit		78.666984			code:0
08:35:10.932674 tree-walk.c:185	d0	main	data_json	r1	78.667065	78.667065	traverse_tre	statistics:{"traverse_trees_count":1,"traverse_trees_max_depth":1}
08:35:10.959060 run-command.c:977	d0	main	child_exit		78.693298	78.575389		[ch0] pid:11500 code:0
08:35:10.959060 trace2/tr2_tgt_perf.c:215	d0	main	atexit		78.693503			code:0
08:35:11.334600 common-main.c:49	d0	main	version					2.36.0.windows.1
08:35:11.342700 common-main.c:50	d0	main	start		0.078582			'C:\Program Files\Git\mingw64\bin\git.exe' rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree --short HEAD
08:35:11.360378 ..._win32_process_info.c:118	d0	main	data_json	r0	0.096641	0.096641	process	windows/ancestry:["bash.exe","bash.exe","bash.exe","bash.exe"]
08:35:11.361039 git.c:460	d0	main	cmd_name					rev-parse (rev-parse)
08:35:11.370327 repository.c:136	d0	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:35:11.374294 wrapper.c:621	d0	main	data	r1	0.110183	0.110183	fsync	fsync/writeout-only:0
08:35:11.374294 wrapper.c:622	d0	main	data	r1	0.110294	0.110294	fsync	fsync/hardware-flush:0
08:35:11.374927 ..._win32_process_info.c:166	d0	main	data_json	r1	0.110906	0.110906	process	windows/memory:{"PageFaultCount":5856,"PeakWorkingSetSize":8847360,"PeakPagefileUsage":5943296}
08:35:11.374927 git.c:719	d0	main	exit		0.110976			code:0
08:35:11.374927 trace2/tr2_tgt_perf.c:215	d0	main	atexit		0.111043			code:0
08:35:11.681099 common-main.c:49	d0	main	version					2.36.0.windows.1
08:35:11.687751 common-main.c:50	d0	main	start		0.078947			'C:\Program Files\Git\mingw64\bin\git.exe' config --bool core.sparseCheckout
08:35:11.706909 ..._win32_process_info.c:118	d0	main	data_json	r0	0.097815	0.097815	process	windows/ancestry:["bash.exe","bash.exe","bash.exe","bash.exe"]
08:35:11.715487 repository.c:136	d0	main	def_repo	r1				worktree:C:/Users/JHatton/Desktop/tmp/8GBi
08:35:11.715844 git.c:460	d0	main	cmd_name					config (config)
08:35:11.717516 wrapper.c:621	d0	main	data	r1	0.108361	0.108361	fsync	fsync/writeout-only:0
08:35:11.717516 wrapper.c:622	d0	main	data	r1	0.108453	0.108453	fsync	fsync/hardware-flush:0
08:35:11.718132 ..._win32_process_info.c:166	d0	main	data_json	r1	0.108938	0.108938	process	windows/memory:{"PageFaultCount":5796,"PeakWorkingSetSize":8814592,"PeakPagefileUsage":4182016}
08:35:11.718132 git.c:719	d0	main	exit		0.109008			code:1
08:35:11.718132 trace2/tr2_tgt_perf.c:215	d0	main	atexit		0.109064			code:1

[jhattongfs]

I went poking through the code and found something that may be helpful. I'm no way qualified to actually attempt to build git though.

Please see this and this. It may or may not be helpful. To me, it looks like a file size of 0 does some special stuff, and any file a multiple of 4GBi will trigger this special stuff.

I can reproduce this issue on Ubuntu 20.04, git 2.25.1, with or without LFS.

[dscho]

Please see this and this. It may or may not be helpful. To me, it looks like a file size of 0 does some special stuff, and any file a multiple of 4GBi will trigger this special stuff.

Oh, that's totally likely the culprit! Because sd_size is of type unsigned int, which is 32-bit. You should therefore even see the same problem with 4GiB files already.

@PhilipOakley started drafting a huge PR to address these type of issues. But that PR only tried to take care of unsigned long instances that really should have been either size_t or off_t. In other words, this unsigned int instance is not even covered.

But wait, it gets worse.

The stat_data data type is used in the cache_entry data type, which in turn is populated from the ondisk_cache_entry data type when reading from Git's index. And in that ondisk_cache_entry data type, size is not even just of type unsigned int, it is specifically uint32_t, i.e. it is locked to 32-bit, and this is an on-disk format that not only git needs to be able to read, but also libgit2, JGit, and every other Git implementation out there. Oh, and git also will have to be able to interact with older Git versions, too.

Therefore, it will be a huge undertaking to fix this. Lucky for me, this is not a Windows-specific problem! Maybe you therefore want to take it to the Git mailing list (send plain-text messages, HTML messages are dropped silently), @jhattongfs?

[rpier1]

' don't know if this is maybe somehow related, but at our company git 2.36.0 is extremely slow (on everything, even git diff).

Largest file I located via this command https://stackoverflow.com/a/46085465/15501748
was 233MiB (So in our case it has nothing to do with 2^33 bytes file.)

What caught my attention is that doing a git diff with trace2 perf and comparing git 2.36.0 with the previous git version...
it seems that fsync lines are now coming in git 2.36.0 (and seems like they might be taking a lot of time). They don't appear in the trace2 perf log, when using an earlier-version of git client... (at least in the brief test I've done).

I see the same fsync stuff in the foldout "git status summary" in this comment above

[rpier1]

What caught my attention is that doing a git diff with trace2 perf and comparing git 2.36.0 with the previous git version... it seems that fsync lines are now coming in git 2.36.0 (and seems like they might be taking a lot of time). They don't appear in the trace2 perf log, when using an earlier-version of git client... (at least in the brief test I've done).

I see the same fsync stuff in the foldout "git status summary" in this comment above

ahh this is probably not related to fsync, since fsync is only reported in new trace2 (in new git release)!
My trace2 run in the old git release cannot show fsync stuff, since it does not yet have that feature: git@9a49876

Nevertheless... the fact remains... the new git 2.36.0 is basically unusably slow for us (we have a huge repository with sparse-checkout).

[PhilipOakley]

@rpier1 Maybe raise a fresh Issue with all the appropriate template details, including which version was 'previous'.

You could try loading the SDK and trying out the different new parts of your git 2.36 (a somewhat directed bisect ;-) to see which aspect is the problem, whether that be the safe.directory CVE fixes, or the fsmonitor updates, or trace2 perf...

Let's leave this ==8GiB file size issue to the specifics of @jhattongfs curated problem.

[PhilipOakley]

@jhattongfs It may be worth taking again to brian @bk2204 as he could advise based on his amending of the entry that follows for the hash algorithm change 575fa8a.

A change from uint32_t to size_t to cover both LP64 and LLP64 systems may be one approach, but one can get bogged down in different views of standards compliance for file (system) size types...

The one other thing to be aware of is that CRC32 (was used on earlier proposals) on parts of the Windows ecosystem only covers up to 2GiB block sizes (because it uses long, rather than size_t, for 'history reasons'), should it still be used for the file.

[bk2204]

If this shows up on the Git list, feel free to CC me at the address I normally use there (consult the .mailmap if you're unsure). I am extremely busy until probably the 10th or 11th due to travel, but I'll probably take a look then (or maybe before, depending) if it shows up there. I have some ideas about how to fix this, but I need to look at this more to see if they're viable.

[jhattongfs]

@bk2204 I sent this into the mailing list last night along with my two cents worth. I'm not entirely sure how the list works, but you can search for this exact title "Git status extremely slow if any file is a multiple of 8GBi" on one of the mirrors. I now realize it should have been a multiple of 4GBi, but too late now :/

[jhattongfs]

I this issue has been resolved upstream and works on git for windows v2.43.0

See commit 5143ac0.