Early EOF errors when running git fetch over ssh #1322
Comments
congyiwu
changed the title
|
bump? |
|
What is the latency between the client and server? I'm willing to take a look at this, but I'd want some sort of repo snapshot (accessible to me) I know can reproduce the issue before diving into setting up a test. |
Based on finding PowerShell/Win32-OpenSSH#1322
|
I think we're seeing this as well. Windows 2016 Server, desktop edition. ssh key is loaded into Windows ssh-agent service. Verbose log output (using public repositories): |
Based on finding PowerShell/Win32-OpenSSH#1322
|
@NoMoreFood I think in our case (based on the timestamps in my log above), around 1.1s between running the command in the client and starting to receive data. |
|
I get this fairly consistently (but not 100% reliably) when cloning Git repos using Win32 SSH. Most commonly when cloning https://github.com/aspnet/AspNetCore |
|
We get this around 50% of the time, intermittently. This also happens with OpenSSH 8.0.0.0p1-Beta |
|
@anurse @petemounce What OSes are you running? |
|
@NoMoreFood I am running Windows Server 2016 Desktop/GUI edition within Google Cloud. Image Family is If it's relevant, https://github.com/GoogleCloudPlatform/compute-image-windows/ is I think what goes into the image-creation process to prepare them for customer usage. |
|
This is the script I'm using to bootstrap sshd onto the instances in question - this is relevant to this issue I think because it's how ssh.exe becomes present, and how the default shell is set to powershell. (The verbatim script relies on GCE-GetMetadata to pull the base64 encoded sshd_config (also snippet'd here) and a base64 encoded public key from GCE instance-metadata. Those sections can be hacked out with hard-coded values if you don't run this within GCE to reproduce; I don't have the Azure/AWS equivalents supported, sorry. I included the full script + sshd_config because I thought you might be interested in adapting it into your docs around enabling pubkey auth - if there's an issue around that, please point me at it and I can replicate there if that's helpful)
|
Windows 10 (various builds, including 18932.1000). Just a standard interactive instance (no docker containers, etc.). Literally just tried and got it again:
|
|
It's not a 100% consistent repro for me, but it's like.. 80-90% consistent. I can usually get a successful clone by repeating a bunch of times. Alternately, if I reset |
|
I badly don't want to use the ssh bundled with git-for-windows because then I (I think?) would need to find a different arrangement for auth, differently from the other two platforms I support. I think my options would be:
|
|
@NoMoreFood friendly ping? Is there any more information I could provide that might help? |
|
@NoMoreFood Running into this on Windows Server 2019 as well (OpenSSH version 7.7.2.1 from Windows Update / pre-installed with Windows). Is there a timeline on addressing this? |
|
@petemounce @RNabel I've been working on other projects, but might be able to sneak this in. Can you verify it's still reproducible on 8.0? |
|
It's definitely still reproducible on 8. I'm less sure it's a data problem, since on retry it can succeed against the same sha being cloned. |
|
I see this all the time when attempting to clone dotnet/roslyn with 7.7.2.1. It's getting to the point that I'm going to have to disable the service. |
|
@NoMoreFood friendly ping? Any news? |
|
If you need a repo that commonly reproduces this, try cloning https://github.com/dotnet/roslyn. It's a pretty reliable failure for me. |
|
I also consistently get this on Windows 10 when running |
|
@NoMoreFood do you need any more information for repro, or want any other help? |
|
Any news on this?I'm seeing the same behaviour... |
|
This seemed to be much better once I moved to |
|
Regarding your "real fix" comment: can you explain a bit more why you think this problem can't be solved properly via stdio and warrants use of a Windows named pipe? Especially in case all that is missing is an fsync() at the end? Since several additional widely-used tools, beyond Git, use ssh via stdio this way, such as Subversion and Rsync, it would be nice if this could be fixed without all these other tools needing new command-line options. After all, PuTTY's plink also appears to manage without that. |
|
Windows doesn't really have stdio in the same way that Unix-like machines do as just boring standard file descriptors, at least as far as I'm aware; there are extra layers involved because they're tied to the windows "console" subsystem. These layers add quirks that unix-y apps try to work around to bring things back to just boring file descriptors. This also gets even more complicated when multiple approaches to this are involved... e.g. git-for-windows uses the mingw layer, while this openssh does not. I took a guess as to what might be happening: that there was still some data still in a buffer somewhere that gets lost when the ssh process exits. I have no idea how to validate this theory to confirm that that's exactly what's happening; it could be that the fsync() perturbs things just enough to make it seem like it works more frequently. On Unix machines, fsync on stdout that's being piped to another process shouldn't have any effect. The named pipe idea just adds an optional path that some apps could decide to implement support for (assuming they don't have a way to read from a file already, because it can just be a |
|
From a glance at |
|
Hm. On POSIX systems, |
|
Correct, this should be a no-op on POSIX systems; the |
|
Does your fsync = FlushFileBuffers call essentially come before OpenSSH for Windows calls ClosePseudoConsole to close the ConPTY? Does this mean that a call to ClosePseudoConsole does not implicitly also do everything that FlushFileBuffers would do? On the ClosePseudoConsole man page, I see this curious remark:
Is all that “draining of information and presenting it” after calling ClosePseudoConsole really done correctly by OpenSSH for Windows? (And if not, is your pre-closure fsync() call just helping to sometimes mitigate the effects of a failure to do that?) Hm ... Looking at contrib/win32/win32compat/win32_pty.c I can't actually see any call to ClosePseudoConsole (and subsequent draining). Is that perhaps the real problem? |
|
Yeah; that's one thing I noticed from |
|
Does CreatePseudoConsole actually happen in conhost.exe (which is called by sshd), and is that what seems to be open-sourced here? But anyway, that's all on the sshd server side, whereas your fsync is on the ssh client, right? |
|
I don't think this goes through the |
|
Bah. Seeing the issue again with either |
|
So another theory of what's happening... git has this code that's doing the reading: It's clearly expecting reads from stdin to be blocking (bytes read == 0 is the error we're seeing). Digging through the Windows I ended up hitting the same "early EOF" issue even while moving comms between the two to a named pipe on windows; the git side is just using the microsoft CRT's implementation of posix file IO though, whereas openssh has its own win32 API layer. |
|
TL; DR: I had same issue on 8.1.p1, uninstalled it and installed Git-2.37.1-64, choosing its bundled ssh, which is OpenSSH_9.0p1, which has resolved it for me. Full post: On Win10, with my .gitconfig containing sshCommand = "ssh -vvv", I'd been using OpenSSH v8.1.p1, and I'd get about 30% failure rate. The command was simple, I just increment NN as 00, 01, 02 etc. Here's the relevant ending output from a failed transfer. Notice the 'fatal: early EOF line'. Having found this thread, I tried a newer OpenSSH. Sadly, OpenSSH V8.9.1.0p1 - same results, roughly 30% failure rate. Then, Retrying my test again exactly as above - perfect run, 20/20 clones now succeed. So, I'm unsure if 9.0p1 has a fix, or if it's simply built different than the win32-openssh versions available here... Either way, I suggest people try out OpenSSH_9.0p1 themselves, and see if it resolves the issue. |
|
If you enable level4 debug output with debug4("write - reporting %d bytes written, io:%p", bytes_copied, pio);and debug4("read - io:%p read: %d remaining: %d", pio, bytes_copied,
pio->read_details.remaining);in contrib/win32/win32compat/fileio.c, which might give clues whether the custom Win32 wrappers for read and write used here do anything inappropriate with zero-length reads and writes, as suggested by @vvuk. |
|
@vvuk Good idea! Looking at fileio_write_wrapper there does not appear to be any protection against a zero-length write happening, which Win32 (unlike POSIX) might pass on the the consumer, that a POSIX application might receive as a zero-length read and interpret as an EOF indication. What happens, if you simply add at the start of that function something like if (bytes_to_copy == 0) return 0;to reduce the likelihood of Or also add debug output in fileio_write to check if Similarly add debug checks in WriteThread in termio.c, to check that the functions |
|
@manojampalam Are you confident that the write calls in fileio.h and termios.h will not accidentally cause zero-byte writes, which the receiving |
so I re-added the Optional Features OpenSSH (8.1.p1), and added this to my global .gitconfig I can still reliably reproduce the issue, but the output doesn't contain any new lines with "write - reporting" or "read - io" in it :-( a recent output: I'm happy to keep experimenting on my 'easy' test case here - lmk what else I could try. |
|
|
|
Hi I believe that I know what is causing this. Why was this issue closed? |
|
When https://github.com/PowerShell/openssh-portable/blob/latestw_all/channels.c#L2051 ultimately dispatches the copying writing to a pipe to a worker thread (this is because it appears that the pipe uses the However the close method introduces several race conditions. If the thread has not yet invoked I have found this issue is not reproducible in a debug build, but is reproducible in a release build. It is not clear to me why Could be moved prior to the invocation of |
|
This is definitely not fixed. |
|
I gave up waiting for Microsoft to fix the bug and ship it — if they’ve failed to fix it for three years and the issue is closed I’m not sure how much clearer they can make the fact they don’t care. I have switched to using PuTTY’s |
|
@LukeCarrier Thanks we also found this solution, but in reality its pretty sad to use another tool, roll it out, update it and so on. In an Enterprise context it's not so nice tbh. And it's pretty sad for MS to just ignore or be unable to simply ship a small bugfix... |
|
For those still following along on this issue, the fix was merged and is available in the current beta: #2012 (comment). |
|
@333fred can you give us any information whether this fix will be backported to Windows 10 given Microsoft's current stance on the Windows 10 update policy? I am asking because Windows 10 is still used in a lot of corporate environments (sometimes not even version 22H2) and it's likely it'll stay this way for a long while. |
|
I have absolutely no idea. I'm just a user who was hit by this bug here 🙂. |
and others
If I configure the Git for Windows client to use the SSH version that ships with Windows, I occasionally see the following error during
git fetch:Here's a log that includes ssh -vvv output:
fetch.log
And here's the test script that I used, which replays the same fetch 100 times in a row:
fetchloop.sh.txt
I believe that this is a bug in the SSH client that ships with Windows, because:
The bug also seems data-shape dependent in that it reproduces more or less frequently depending on the exact state of the Git repos on the client and server. I have snapshots of the client and server Git repo state. I can't share it publicly, can can share it internally with Microsoft employees.
My client OS is Windows 10 1809 Build 17763.253
((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion)says 7.7.2.2The Git servers that I have tested against are:
From messing around with the Azure DevOps SSH code, I found the following:
So this looks like a race condition where channel data from the server is lost when the client receives SSH_MSG_CHANNEL_CLOSE.
BTW, I tried searching for related issues and found #752 and a few others, but they all seemed to be issues with running the Windows version of sshd on the server, as opposed to the client.
@dscho this should probably block git-for-windows/git#1981
The text was updated successfully, but these errors were encountered: