Git Bash significantly slower after upgrade to 2.41.0

[sherman89]

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

64-bit version of Git 2.41.0.windows.1

I already downgraded to 2.40.1, apologies.

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows 11

Microsoft Windows [Version 10.0.22621.1778]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: Notepad++
Custom Editor Path: 
Default Branch Option:  
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

No

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash on Windows Terminal

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

None, just upgraded to 2.41.0.

• What did you expect to occur after running these commands?

Same performance as before.

• What actually happened instead?

Everything is slower, cd to directory, git pull, git log, literally everything is significantly slower in Git Bash.

Every upgrade I did before throughout the years has been fine, except this one.

Downgrading to 2.40.1 fixed the issue.

[quyenlv-unicloud]

+1, it take 30s to start in my computer. Promon show new version just waste 10s for query "C:\Program Files\Git\usr\bin\DNSAPI.dll" and got "NAME NOT FOUND",

while old query "C:\Windows\System32\dnsapi.dll" and immediately got "SUCCESS"

[mormegil-cz]

Same here, the problem is with the bundled GNU utils. Git itself is fine and some built-in functionalities work normally, e.g. git status. However, any GNU tool takes several seconds to run. Just running e.g. "c:\Program Files\Git\usr\bin\arch.exe" takes ~20 seconds. So any git command running a bash subshell or less or anything takes very long time to execute.

P.S. Downgrade to 2.40.1 fixes the problem, while even 2.41.0-rc0 is broken.

[bricss]

Surprisingly, I cannot confirm any of this ❌
The latest Git version is blazingly fast ⚡ on both of my 2️⃣ machines with the latest Win11 x64 (OS build: 22621.1778)

[Osyotr]

It's slow on Windows 10 but OK on Windows 11.

[mormegil-cz]

For reasons unknown to me, when running even just true.exe, it seems to run LDAP queries against the Windows domain controller. Which seem to be correlated to the startup delay (but dunno, it might be coincidence).

(I don’t have symbols for msys-2.0.dll.)

Which… might be an upstream issue in msys2 maybe? (And it might also be caused by our DC being misconfigured or something, but still… I don’t think true.exe needs to depend on proper LDAP configuration…)

[razvan-pricope]

We experience the same issue in our organization. The common thing for all of the employees is that the computers experiencing the slowdown are domain joined. Maybe this is the reason it's not 100% reproducible.

[PhilipOakley]

Domain controller delay problems show up moderately often. Not easy to resolve. Worth searching the archives (closed issues ;-)

[mormegil-cz]

OK, running

mkpasswd -l -c > passwd
mkgroup -l -c > group

in elevated shell in C:\Program Files\Git\etc seems to have helped. Still, no idea what changed in the 2.41.0 release which prompted the problem (and what the real trouble with (our?) DC/LDAP is, anyway).

[mormegil-cz]

OK, that did not really work, since the created passwd file had home directories pointing to /home/username which is wrong (unix-y), it needs to point to /c/Users/username so that ssh works; I had to modify the path manually. (Without it, git fetch did not find known_hosts and even when I confirmed the fingerprint, it failed with “Failed to add the host to the list of known hosts”.)

[MikeCheel]

git branch command takes excruciatingly long (i.e. minutes) to return. Downgrading to previous version of git solves the problem. Also, not just git bash. This occurs in any terminal.

[rglidden]

Having similar performance issues on Windows 11 under PowerShell (not using git bash). Some commands like git status are fine, but other commands like git branch and git fetch take forever. Large internal repos hosted in our on-prem Azure DevOps are excruciatingly slow. Smaller Github-hosted repos are a little better, but still noticeably slower. Machine is domain-joined. Have not yet tested on a non-domain joined machine.

Reverting to 2.40.1 restored all performance back to acceptable levels.

EDIT: I was able to test 2.41 on a non-domain joined machine, and I did not experience the same slowdowns. As others said, it appears to be domain related.

[belucha]

I had quite similar problems, only when connected with the domain.

It was totally unrelated to the repository, running
git config --global --list was taking about 15seconds to complete, whenever I was connected to domain.

Reverting back to 2.40.0 resolved all issues.

I turned on all possible traces GIT_TRACE=1,GIT_TRACE_SETUP,GIT_TRACE_PERFORMANCE and found that the delay was related, whenever a sub process was run by git. I was able to see the exec-command and run-command trace output, and then a large delay of 15s.

I case of the above git config global --list the delay was caused by calling the pager less. I speed this command up setting GIT_PAGER to an empty string, but the root cause seems to be similar to the above described problems.

Unfortunately I was not able to identify the problematic network request.

[geniuscodemonkey]

I'm seeing a git pull taking over 4 minutes when it was taking a couple of seconds. Half the time GIT will also crash... (I can't upload an image due to work security policy)
I've had to downgrade git to 2.39

[MikeCheel]

This problem is happening in Windows 10 using powershell as well. I can further confirm that, as mentioned in this discussion previously, the gnu utils seem to be the culprit. Running the command "c:\Program Files\Git\usr\bin\arch.exe" to around 30 seconds to return. Running under admin prompt makes no difference. UPDATE: I noticed today (25 July) when I am connected to my company network the problem happens but when not connected everything works normally. The cause is related to this somehow.

[dscho]

I cannot reproduce here. For those who can, do you see the same issue with regular Cygwin (on which Git's Bash is based)?

[ianichitei]

I cannot reproduce here. For those who can, do you see the same issue with regular Cygwin (on which Git's Bash is based)?

Using cygwin 3.4.6.1 and simply running any .exe from that archive takes more than 20 seconds:

$ Measure-Command { .\ldd.exe --help }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 22
Milliseconds      : 552
Ticks             : 225529747
TotalDays         : 0.000261029799768519
TotalHours        : 0.00626471519444444
TotalMinutes      : 0.375882911666667
TotalSeconds      : 22.5529747
TotalMilliseconds : 22552.9747
$ .\ldd.exe --version
ldd (cygwin) 3.4.6
Print shared library dependencies
Copyright (C) 2009 - 2023 Chris Faylor
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[dscho]

@ianichitei thank you. There is Cygwin v3.4.7 available now which might fix the issue. If not, could I ask you to report the bug to the Cygwin project? They probably can find a fix relatively quickly given your assistance, and we can then backport the fix into Git for Windows.

[ianichitei]

I tried with 3.4.7 and the issue is not fixed. I'll try to report this directly to the Cygwin devs.

[ianichitei]

Here is the Cygwin bug report.

[ianichitei]

Based on the responses from the Cygwin mailing list I tried the second workaround from here and it solved the problem for me: Cygwin Terminal now starts instantly. I don't know how one could do something similar for the Cygwin bundled with Git for Windows.

Also, while the FAQ mentions that "since Cygwin 1.7.34, new installations now use the SAM/AD databases directly", is it possible that the previous Git for Windows version did not had this turned on? It is a bit weird that it started to happen to different people from different organizations after updating to 2.41.0.

[dscho]

I tried the second workaround from here and it solved the problem for me

You mean this?

Run these commands in a Cygwin terminal with write access to /etc:

getent passwd $(id -u) > /etc/passwd
getent group $(id -G) > /etc/group

Also, set /etc/nsswitch.conf as follows:

passwd: files db
group:  files db

If so, we already have the latter (even more than that: we comment out the db part in the group line). But the former only works in single-user setups, which Git for Windows does not want to be limited to.

It is a bit weird that it started to happen to different people from different organizations after updating to 2.41.0.

Indeed. The only thing we did that is notable was to upgrade from MSYS2 runtime v3.3.* to v3.4.*. But there were so many changes between those major versions that I would suspect some regression to have crept in.

[ianichitei]

You mean this? [...] If so, we already have the latter (even more than that: we comment out the db part in the group line). But the former only works in single-user setups, which Git for Windows does not want to be limited to.

Yes, exactly those, but if that is already done in case of Git it probably means that there's another cause for the problem, and the slowdown I experienced with Cygwin has nothing to do with the one experienced with Git for Windows.

Indeed. The only thing we did that is notable was to upgrade from MSYS2 runtime v3.3.* to v3.4.*. But there were so many changes between those major versions that I would suspect some regression to have crept in.

I don't really know how to further reply to the Cygwin issue with regards to this, but maybe someone on your side could add some context to that bug report?

[mormegil-cz]

Yes, exactly those, but if that is already done in case of Git it probably means that there's another cause for the problem, and the slowdown I experienced with Cygwin has nothing to do with the one experienced with Git for Windows.

As I mentioned above, filling the C:\Program Files\Git\etc\passwd file did help for me; so yes, this seems to be relevant; however, whose “fault” this is and what is the correct generic solution, I have no idea.

[dscho]

filling the C:\Program Files\Git\etc\passwd file did help for me; so yes, this seems to be relevant;

Indeed. This basically provides the information about the current user such as the domain, the user name, and the SID.

I bet we could obtain that information about the current user much quicker than going through Active Directory calls. However, debugging such things in the MSYS2 runtime is always quite a pain, in particular when you lack a setup to reproduce the issue :-(

[jankap]

I have the same issues at my corporate workstation. As long as I'm connected to our VPN (which is required for most workflows), Git is painfully slow. git config --global --list is slow, git status is slow, git commit --amend (which calls VSCode) is slow. Disabling the VPN and everything is snappy again.

How can I help to debug that?

Edit: looks like I'm on msys, not cygwin:

$ bash --version
GNU bash, version 5.2.15(1)-release (x86_64-pc-msys)

Disabling the VPN still helps.

[dscho]

looks like I'm on msys, not cygwin:

You're actually on MSYS2, and both MSys and MSYS2 are forks of Cygwin, so for the purposes of this ticket it is irrelevant on which system you are.

Disabling the VPN most likely cuts off the connection to the domain controller and the slow calls are simply skipped, which is consistent with what we knew already.

What we still don't know is which parts of Cygwin's code is executed in the slow path, and most crucially: why it is executed when we do not really need all that information that the domain controller is asked to report.

What would be helpful, therefore, would be a thorough debugging session that most likely requires instrumenting (and building) the MSYS2 runtime and repeatedly running simple commands both with the VPN enabled and disabled to identify above-mentioned code paths.

[rglidden]

I spent some time looking through diffs between MSYS2 3.3 and MSYS2 3.4, and I suspect the issue is related to git-for-windows/msys2-runtime@a5bcfe6 which removes a short-circuit for the current user and instead reaches out to AD for user information. This might also explain why having the "db" portion of the groups line in /etc/nsswitch.conf commented out doesn't prevent this problem from occurring.

I don't really have the experience necessary to figure out how to build MSYS2 and get Git to use it to test it out quickly. I may use it as a learning experience, but just wanted to point this out in case anyone more experienced with building MSYS2 and git can give it a quick test, or provide me with some test binaries to try.

[dscho]

I suspect the issue is related to git-for-windows/msys2-runtime@a5bcfe6 which removes a short-circuit for the current user and instead reaches out to AD for user information.

Great work, @rglidden, that indeed looks like a plausible culprit!

I don't really have the experience necessary to figure out how to build MSYS2 and get Git to use it to test it out quickly.

I'm happy to report that it's not all that difficult to get started, in particular because we have tons of automation by now.

All you've got to do is to fork & clone https://github.com/git-for-windows/msys2-runtime/, git revert that commit locally and then open a PR. This will trigger an automated build in the "Checks" tab, called "build on: pull_request" (example). Once the build is done, you can click on that word "build", which is a link and will lead you to the summary where you can see the install build artifact (example). This artifact is a .zip file that contains, among other files, usr\bin\msys-2.0.dll, which you can drop into, say, an unpacked PortableGit (e.g. v2.42.0), overwriting the existing usr\bin\msys-2.0.dll. Then you can call git-bash.exe from that PortableGit's top-level directory to test.

If this is indeed the culprit, I'll help you introduce a new setting for the nsswitch.conf that we will use in Git for Windows from now on, to enable the fast path again.

[rglidden]

Thanks for the detailed instructions @dscho. I was trying to figure out how to build things locally. I didn't realize I could leverage your build infrastructure by submitting a PR. That will make things a LOT easier!

I'll see if I can find some time to try it in the next couple days and will let you know what I find.

[dscho]

Excellent!

[rglidden]

Looks like my hunch was correct!

1. Verified that PortableGit-2.42.0 still suffers the performance problem on my machine. ~30 seconds to launch git bash or do most git commands.
2. Copied the msys-2.0.dll from msys2-runtime: restore fast path for current user primary group msys2-runtime#57 into usr/bin/msys-2.0.dll in my PortableGit folder.
3. Problem is resolved. Git bash loads nearly instantly, and all git commands are now responsive.

[dscho]

@rglidden awesome work, and I am so glad that your hunch was correct and now you have proof!

So, let me walk you through the process how to add a keyword to /etc/nsswitch.conf and how to let that keyword toggle how the code behaves.

The central file to do this, the file where that db keyword is parsed, is winsup/cygwin/uinfo.cpp. You can see that the NSS_SRC_DB flag is set when db is matched. Whether this NSS_SRC_DB flag is set or not happens to be queried exclusively via nss_pwd_db(), most notably in internal_getpwsid() and in internal_getpwnam().

You will probably realize that despite passwd: files db suggesting an order, suggestive of a precedence of "files" over "db", the precedence is the same if you write passwd: db files (still "files" over "db") because the values are parsed into the bit field pwd_src, and the code will always first query nss_pwd_files() and only when that did not find anything the code goes on to the if (nss_pwd_db()) clause.

This is good! Because it means that we do not have to deal with precedence other than the hard-coded one. We can even add a keyword, say, db-quick-and-dirty (I did not really find a word reflecting what we want to do here, so I went with my old band's name) that includes the meaning of db, by letting it set NSS_SRC_DB and another flag, say, NSS_SRC_DERIVE_GROUP_FROM_CURRENT_USER_TOKEN that we can then use in fetch_account_from_windows() to reinstate the fast code path.

But then, we probably want to default to the fast 'n furious code path in the MSYS2 runtime, and only do the correct-yet-slow work if asked to do so, as we all do? And while I talked about passwd all the time, I now realize that the group: files db line is a much more appropriate target for this here discussion: after all, the problematic slow-down happens because we want to fetch the current user's default group, and the fast path simply gets it from the current user token, which is sometimes incorrect.

So here is what I'd suggest to do:

• Amend the commit git-for-windows/msys2-runtime@a51cbf8 with the following:
• Add a new NSS_SRC_ACCURATE = 4 constant next to NSS_SRC_FILES and NSS_SRC_DB.
• Introduce a new group keyword in the /etc/nsswitch.conf file, say, db-accurate, and parse it like db, but assign *src |= NSS_SRC_DB | NSS_SRC_ACCURATE. Maybe print a warning if the keyword is used in a passwd line because it won't have any effect. If you prefer, you can also name it accurate and only assign *src |= NSS_SRC_ACCURATE, but that has the downside that a group: files accurate line will be valid but the accurate won't do anything, which might surprise some.
• Add a new nss_grp_accurate() method next to nss_grp_files() and nss_grp_db().
• Rename the is_current_user variable to something more meaningful, e.g. get_default_group_from_current_user_token.
• Augment the sid == cygheap->user.sid () condition guarding the fast path using the newly-introduced nss_grp_accurate() method, i.e. if (!cygheap->pg.nss_grp_accurate () && sid == cygheap->user.sid ()).
• Document the new keyword in the appropriate section, taking care to mention that it only has an effect on the group line. Use the information from the commit message of git-for-windows/msys2-runtime@a5bcfe6 to illustrate why users may want to use that flag.
• Write a kick-ass commit message to motivate all of this, mentioning that "This fixes Git Bash significantly slower after upgrade to 2.41.0 #4459".

Questions? 😁

[clach04]

This is a me-too comment :-D

1. I have a machine not on a domain, with 2.31.0 which is fine.
2. I have a machine on a domain, tested with 2.41.0.3-64-bit and latest (as of 2023-08-25) Git-2.42.0-64-bit - which is slow (for diff).

Originally I thought #4574 was different to this issue #4459 but I'm now suspecting they are the same based on the pager test suggested in that issue.

[rglidden]

Thanks again for all the extremely detailed instructions @dscho! This has been a very helpful and welcoming introduction to submitting changes to git for Windows. 😄

I've updated git-for-windows/msys2-runtime#57 with the changes you suggested. Tested locally and it resolves the performance issues, while preserving the more "correct" behaviour by adding the group: db-accurate option.

Please review when you can at let me know if anything needs adjusting.

[dscho]

@rglidden thank you so much! I am in the process of integrating this into Git for Windows (will probably have to take a break and finish it tomorrow).

[sherman89]

@rglidden Thank you very much! :)