Git submodule update --recursive --remote --init freezes after a few seconds

[Vincent-Liem]

Existing issues matching what you're seeing

• I was not able to find an open or closed issue matching what I'm seeing

Git for Windows version

git version 2.50.0.windows.1
cpu: x86_64
built from commit: 311e67dc2479debe725dfb9a3f974bdce04644ea
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.14.1
OpenSSL: OpenSSL 3.2.4 11 Feb 2025
zlib: 1.3.1
SHA-1: SHA1_DC
SHA-256: SHA256_BLK

Windows version

Windows 10

Windows CPU architecture

x86_64 (64-bit)

Additional Windows version information

Microsoft Windows [Version 10.0.19045.5965]

Options set during installation

Editor Option: VIM
Custom Editor Path:
Default Branch Option: main
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable FSMonitor: Disabled

Other interesting things

No response

Terminal/shell

Git Bash

Commands that trigger the issue

git init
git submodule add git@github.com:<username>/repository
git submodule update --recursive --init --remote --progress

Expected behaviour

The nested submodules should be updated to the latest commit.

Actual behaviour

The progress bar freezes randomly after a few seconds, with no sign of any download progress despite good internet connection. This only happens on the newest version. Downgrading to 2.49.0.windows.1 fixes the issue immediately. The difference with issue #5663 is in the version where this bug happens.

Repository

Latest version (2.50.0.windows.1)

[dscho]

Could you please find out which snapshot is the last good one?

[paaaz]

Same probleme here with Windows 11... for some seconds full download speed of ssh.exe, then the rate drops to some kbps and then to 0.
Our IT department just blocked rolling out the update to v2.50.0, seems to be a serious issue!

[dscho]

@paaaz could you please find out which snapshot is the last good one?

[NaterGator]

I am running into this issue as well, exact same symptoms as described.
@dscho here are the results of my testing:

• 311e67dc2479debe725dfb9a3f974bdce04644ea is affected
• b982220acf856d04cfd8238db5a10cd93f36d89a is affected
• 7e61ebc1ebe938453f999949b39e8bf134e43a4b is affected
• bb56068abb5d793388c050d598e01e7dd505f467 is affected
• e0046ebc2e645f37204f9e8dcb8286b0129d1cb9 is affected
• 92f4ffb0101d8a46c85f69eded8d98105ec283b7 is affected
• 773e41c596e107769f2504f4dd6501f961f9f543 is affected
• 4ca71ba5311a8f1bafbf002e97e076f15dcfc15b is affected
• 357363f92fb6c05732c26bd6f3268e148d8a62b7 is not affected

Given this is the diff between those two, I thought maybe the problem was trying to clone on my ReFS VHDX, so I reinstalled 2.50.0.windows.1 and tried the same clone on an NTFS drive but hit the exact same issue.

[dscho]

@NaterGator thank you for testing. Looking at the /etc/package-versions.txt files included in the respective PortableGits (2.49.0.windows.1-7-g4ca71ba531-20250326122305 and 2.49.0.windows.1-5-g357363f92f-20250319123930), I do not see anything obvious (I half expected an openssh update to have been the culprit):

--- /dev/fd/62	2025-06-19 17:29:20.000000000 +0200
+++ /dev/fd/61	2025-06-19 17:29:20.000000000 +0200
@@ -9,7 +9,7 @@
 docx2txt 1.4-2
 dos2unix 7.5.2-1
 expat 2.7.0-1
-file 5.46-2
+file 5.46-1
 findutils 4.10.0-2
 gcc-libs 13.3.0-1
 git-flow 1.12.3-1
@@ -19,7 +19,7 @@
 gzip 1.13-1
 heimdal-libs 7.8.0-5
 less 668-1
-libassuan 3.0.2-1
+libassuan 2.5.7-1
 libbz2 1.0.8-4
 libcbor 0.11.0-1
 libcurl 8.12.1-2
@@ -32,7 +32,7 @@
 libgpg-error 1.51-1
 libhogweed 3.10.1-1
 libiconv 1.18-1
-libidn2 2.3.8-1
+libidn2 2.3.7-1
 libintl 0.22.5-1
 libksba 1.6.7-1
 liblz4 1.10.0-1
@@ -69,24 +69,24 @@
 mingw-w64-x86_64-expat 2.7.0-1
 mingw-w64-x86_64-gcc-libs 14.2.0-3
 mingw-w64-x86_64-gettext-runtime 0.24-1
-mingw-w64-x86_64-git 2.49.0.windows.1.7.g4ca71ba531.20250326122305-1
+mingw-w64-x86_64-git 2.49.0.windows.1.5.g357363f92f.20250319123930-1
 mingw-w64-x86_64-git-credential-manager 2.6.1-1
-mingw-w64-x86_64-git-doc-html 2.49.0.windows.1.7.g4ca71ba531.20250326122305-1
+mingw-w64-x86_64-git-doc-html 2.49.0.windows.1.5.g357363f92f.20250319123930-1
 mingw-w64-x86_64-git-extra 1.1.653.48e2403b3-1
 mingw-w64-x86_64-git-lfs 3.6.1-1
 mingw-w64-x86_64-gmp 6.3.0-2
 mingw-w64-x86_64-gnutls 3.8.4-1
 mingw-w64-x86_64-libffi 3.4.7-1
 mingw-w64-x86_64-libiconv 1.18-1
-mingw-w64-x86_64-libidn2 2.3.8-1
+mingw-w64-x86_64-libidn2 2.3.7-2
 mingw-w64-x86_64-libpsl 0.21.5-3
 mingw-w64-x86_64-libssh2-wincng 1.11.1-1
 mingw-w64-x86_64-libssh2-wincng 1.11.1-1
-mingw-w64-x86_64-libsystre 1.0.2-1
+mingw-w64-x86_64-libsystre 1.0.1-6
 mingw-w64-x86_64-libtasn1 4.20.0-1
 mingw-w64-x86_64-libtre 0.9.0-1
 mingw-w64-x86_64-libunistring 1.3-1
-mingw-w64-x86_64-libwinpthread-git 12.0.0.r619.g850703ae4-1
+mingw-w64-x86_64-libwinpthread-git 12.0.0.r576.g49111ba98-1
 mingw-w64-x86_64-libzip 1.11.3-1
 mingw-w64-x86_64-nettle 3.10.1-1
 mingw-w64-x86_64-nghttp2 1.65.0-1
@@ -100,9 +100,9 @@
 mingw-w64-x86_64-xz 5.6.4-1
 mingw-w64-x86_64-zlib 1.3.1-1
 mingw-w64-x86_64-zstd 1.5.7-1
-mintty 1~3.7.8-1
-mpfr 4.2.2-1
-msys2-runtime 3.6.0-1
+mintty 1~3.7.7-1
+mpfr 4.2.1-1
+msys2-runtime 3.5.7-3
 nano 8.3-1
 ncurses 6.5.20240831-2
 nettle 3.10.1-1
@@ -141,7 +141,7 @@
 perl-WWW-RobotRules 6.02-3
 perl-http-cookiejar 0.014-1
 perl-libwww 6.78-1
-pinentry 1.3.1-2
+pinentry 1.3.1-1
 sed 4.9-1
 bash 5.2.037-1
 ssh-pageant 1.4.r15.gc804ba4-1

It might be the MSYS2 runtime upgrade, though. Could you see whether copying /usr/bin/msys-2.0.dll from 2.49.0.windows.1-5-g357363f92f-20250319123930 into 2.49.0.windows.1-7-g4ca71ba531-20250326122305 "fixes" it? (Assuming that the bug reproduces in a Portable Git, that should be relatively quick to test without affecting your installed Git for Windows.)

[NaterGator]

@dscho you were right on the money.
4ca71ba with the msys-2.0.dll from 357363f cloned no issues on the first try. To verify I swapped back to the original msys version from 4ca71ba and it again failed on the first try.

[dscho]

Okay, so now comes the daunting task of figuring out which of the 445 commits in cygwin/cygwin@cygwin-3.5.7...cygwin-3.6.0 broke the thing :-(

I am technically off of work for the rest of the week, so I will be unlikely to get to dig into this in any meaningful in the next few days.

This is what I would do: use git bisect to pinpoint the actual commit.

The big challenge is not only that building the MSYS2 runtime requires a full Git for Windows SDK, and takes a while, and then the result has to be tested in a Portable Git because you cannot overwrite the msys-2.0.dll as long as it is in use, but first and foremost that MSYS2 (and also Git for Windows) put a few changes on top of the commits in https://github.com/cygwin/cygwin.

I did bisect a couple of times in the past, basically by squashing all of those MSYS2/Git for Windows changes into one single, big commit, and then calling git cherry-pick -n <commit> before building each revision during the git bisect run. Of course that big commit does not necessarily apply throughout that commit history, so yes, it's a bit tedious.

@NaterGator I will not ask you to do this, but if you're up to the challenge, this is what I would recommend to do.

[NaterGator]

I'll see how far I can get bisecting the 445 commits after I'm off work for the day 😉 Hopefully I can repro on my personal Ryzen 9950X3D machine because it builds quite a bit faster than my work laptop.

[orgads]

Does it make sense to make a new release with downgraded msys2 until this is solved?

[dscho]

Does it make sense to make a new release with downgraded msys2 until this is solved?

Not really. There have been tons of bugfixes in that MSYS2 runtime.

It makes more sense for affected users to downgrade while we figure out a proper fix.

And it's not like the majority of users clone via SSH; HTTPS is the most common protocol in use for cloning/fetching/pushing.

[NaterGator]

@dscho I got started this AM getting the SDK installed and attempting an initial build. I've run into a bunch of compilation errors after spawning an MSYS shell via msys2_shell.cmd and running sdk build msys2-runtime which I suspect is down to me building based on everything checked out at defaults:

 CXX      init.o
In file included from /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygtls.h:300,
                 from ./globals.h:5,
                 from /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/winsup.h:281,
                 from /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/advapi32.cc:9:
/usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/ntdll.h:492:16: error: redefinition of 'struct _FILE_STAT_INFORMATION'
  492 | typedef struct _FILE_STAT_INFORMATION           // 68
      |                ^~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/w32api/minwindef.h:163,
                 from /usr/include/w32api/windef.h:9,
                 from /usr/include/w32api/windows.h:69,
                 from /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/winlean.h:58,
                 from /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/winsup.h:82:

Apologies for being dense; I was hoping to baseline an initial build from a release tag like 2.50.0.windows.1 just to confirm everything is copacetic with my environment before I start bisecting. Is there an sdk script that automates checking out the dependent projects to known tags or commitrefs that were used to generate a released build? Or is my best bet to look at /etc/package-versions.txt from the portable installer for a specific version and go check out tags / commitrefs listed there?

Edit: looks like msys2-w32api-headers hasn't yet included this patch which fixes the issue. Had to manually apply a couple of patches to support building on msys2 v13 but I've got a build working starting at 2644508f so I should be able to start bisecting. Will let ya know what I find

So far I know the issue was introduced prior to 9a4a4452

[NaterGator]

Eureka! Thanks for the pointers @dscho

nweib@NatesPC MSYS /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime ((c607889824...)|BISECTING)
$ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[7ed9adb356df61114bb3fa3d0efc098c0501921c] Cygwin: pipe: Switch pipe mode to blocking mode by default

Looks like 7ed9adb3 is the culprit, which was implemented to fix #5115

cc: @tyan0: does anything obvious jump out to you regarding this hang?

[dscho]

@NaterGator great finding! I'll refer to 7ed9adb3 as cygwin-3.6.0~226 in the following (or short, ~226).

In the meantime, I could also verify that this affects plain Cygwin: running git -c core.sshCommand=/d/cygwin64/bin/ssh.exe clone git@github.com:SFML/SFML (when you have a Cygwin installation at D:\cygwin64) reproduces the problem.

I tried to rebase the revert of ~226 to cygwin-3.6.0 but got stuck at ~22 which seems to rely on ~226 not being reverted. Of course, even if that would have succeeded, it would have been quite crappy to re-break #5115.

Sadly, ~226 is nowhere close to obvious, even including the commit message. It's never a good sign when GitHub displays "Large diffs are not rendered by default." for a bug fix commit.

Therefore I am unable to help any further with any reasonable amount of work, and have reached out to @tyan0 via IRC, asking him to fix the regression.

[dscho]

Unfortunately, @tyan0 indicated on IRC that they are busy with another problem and will only have a chance to look at this issue later.

So here are a couple of findings:

First of all, the stacktrace when attaching to that stuck ssh.exe process looks like this:

#0  0x00007ff90bc01d24 in ntdll!ZwQueryInformationFile () from /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ff8340ba5be in fhandler_pipe_fifo::raw_write (len=2097152, ptr=0x6fffffbf1ff0, this=0x8000238b0)
    at ../../.././winsup/cygwin/fhandler/pipe.cc:490
#2  fhandler_pipe_fifo::raw_write (this=0x8000238b0, ptr=0x6fffffbf1ff0, len=2097152) at ../../.././winsup/cygwin/fhandler/pipe.cc:443
#3  0x00007ff834062d41 in write (fd=6, ptr=0x6fffffbf1ff0, len=2097152) at /usr/src/cygwin-runtime/winsup/cygwin/local_includes/dtable.h:64
#4  0x00007ff834127ef4 in _sigfe () at sigfe.s:35
#5  0x0000000100442365 in ?? ()
#6  0x0000000100444a23 in ?? ()
#7  0x00000001004170c6 in ?? ()
#8  0x000000010049e53b in ?? ()
#9  0x00007ff833fd8476 in dll_crt0_1 () at ../../.././winsup/cygwin/dcrt0.cc:1003
#10 0x00007ff833fd5f03 in _cygtls::call2 (this=0x7ffffce00, func=0x7ff833fd7370 <dll_crt0_1(void*)>, arg=0x0, buf=buf@entry=0x7ffffcdf0)
    at ../../.././winsup/cygwin/cygtls.cc:41
#11 0x00007ff833fd5fc5 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at ../../.././winsup/cygwin/cygtls.cc:28
#12 0x0000000000000000 in ?? ()

This is thread 1, the other 6 threads are uninteresting.

When I then let gdb continue (with the c command), I can no longer interrupt the program using Ctrl+C. When I then terminate the gdb process (via Stop-Process in a separate PowerShell window) and then try to attach again, the entire window is closed (suggesting serious Console corruption is going on).

Will keep on digging.