SSH to remote causes disconnect with client_loop: send disconnect: Broken pipe #7966

rochecompaan · 2022-01-24T17:42:43Z

Version

Microsoft Windows [Version 10.0.22000.434]

WSL Version

WSL 2
WSL 1

Kernel Version

5.10.60.1

Distro Version

Ubuntu 20.04

Other Software

No response

Repro Steps

ssh to any remote machine from Ubuntu 20.04 under WSL. ssh from Powershell works without issues.

Expected Behavior

ssh connection should persist and stay connected

Actual Behavior

Consistent disconnects after a few seconds:

$ ssh upfront4                                                                                                                  bash  19:28:08 
Linux upfront4 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have new mail.
Last login: Mon Jan 24 13:19:23 2022 from 102.39.132.97
roche@upfront4:~$ sudo -i
[sudo] password for roche: client_loop: send disconnect: Broken pipe

Diagnostic Logs

networking.bat.log
wsl.etl.zip

The text was updated successfully, but these errors were encountered:

eqn-group · 2022-01-25T14:08:07Z

read answer here: 616355

for per-user configuration, Edit the file ~/.ssh/config to have:

Host *
            ServerAliveInterval 60
            TCPKeepAlive no

this could be also set globally in /etc/ssh/ssh_config. This issue is not related to WSL.

ServerAliveInterval: number of seconds that the client will wait before sending a null packet to the server (to keep the connection alive).

rochecompaan · 2022-01-25T16:50:14Z

I already had the following in ~/.ssh/config before logging this issue:

Host *
ServerAliveInterval 60

I just added TCPKeepAlive no and I still hit the same error. If it is not a WSL issue, why don't I experience the same issue when using ssh under Windows?

rochecompaan · 2022-01-25T16:57:52Z

change ServerAliveInterval to 20

Looks promising, thanks! I'll do a few more tests and confirm that the error is gone.

rochecompaan · 2022-01-26T08:46:53Z

Unfortunately, the issue is not resolved. SSH aside, if I run side-by-side pings from Windows and WSL to a remote host, I see up to 20% packet loss on the WSL network, but 0% on Windows.

rochecompaan · 2022-01-26T08:56:38Z

My issue seems similar to #7326 but updating the name server in resolv.conf does not resolve my issue. Other issues that seem related are #7254 and #6416.

eqn-group · 2022-01-26T09:03:42Z

You can try in debug mode with command ssh -v root@server.ip 2>result.txt and share the debug log here.
also did you try pinging other servers ?

rochecompaan · 2022-01-26T10:06:29Z

ssh client:

dpkg -l | grep ssh-client
ii  openssh-client                       1:8.2p1-4ubuntu0.4                    amd64        secure shell (SSH) client, for secure access to remote machines

I manage clusters of servers on multiple continents and I hit the same issue no matter what server I ping or ssh.
result.txt

I honestly fail to see how this is an ssh issue, more likely an issue with TCP NAT again: #6416 (comment)

victorpablosceruelo · 2022-01-26T14:46:18Z

Hi there.
I'm having the same issues.

My log with verbose:

vpablos@WKS0001108274:~$ ssh -v -o TCPKeepAlive=yes -o ServerAliveCountMax=20 -o ServerAliveInterval=15 localhost
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/vpablos/.ssh/id_rsa type 0
debug1: identity file /home/vpablos/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to localhost:22 as 'vpablos'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:41liDTj8s2Y10brK8MN63e8eoMVs3q0GrKc/AVk+RDE
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/vpablos/.ssh/known_hosts:6
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/vpablos/.ssh/id_rsa RSA SHA256:pvyMfM8ata4DoXBG9jnPkTUHqrQIFoVBxGaS9L0sf0U agent
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /home/vpablos/.ssh/id_rsa RSA SHA256:pvyMfM8ata4DoXBG9jnPkTUHqrQIFoVBxGaS9L0sf0U agent
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to localhost ([127.0.0.1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
client_loop: send disconnect: Broken pipe
vpablos@WKS0001108274:~$

rochecompaan · 2022-01-26T17:21:57Z

@Equation-Group can you explain why this would make a difference? We don't permit root logins on any of our servers in any event. I'm seeing disconnects on almost any utility or script that uses the network so this is clearly not an ssh issue.

RandolphBack · 2022-01-27T17:13:53Z

I was having the same problem setting up an sftp user. Every connection by sftp was immediately disconnected
with the "Broken pipe" message as above.

I discovered this message in the /var/log/message
fatal: bad ownership or modes for chroot directory "/srv/sftp/data/ftpuser"

The sshd_config directory specified that "/srv/sftp/data/ftpuser" in the ChrootDirectory.

The ftpuser directory was owned by ftpuser.
Changed that to be owned by root and created a subdirectory as an upload directory owned by ftpuser.

Now sftp connections are working.

eqn-group · 2022-01-30T21:49:54Z

@rochecompaan @RandolphBack The fix is change MTU to 1350 using cmd netsh interface ipv4 set subinterface "vEthernet (WSL)" mtu=1350 store=persistent

and then do this: #4901 (comment)

rochecompaan · 2022-01-31T09:22:46Z

@Equation-Group changing MTU or DNS settings does not work for me. I saw this comment explaining how to change "Auto metric" on the TCP/IP settings for the adapter to "1" and that seems to work! I will log ping stats for a few days to confirm.

rochecompaan · 2022-01-31T14:04:47Z

I have been running a ping script for several hours now, and I do not see any packet loss, so this is resolved for me.

I just noticed that I have 2 WiFi adapters, which was probably causing the issue. After reading up on automatic metric feature , I guess setting the interface metric to 1 on one of the adapters forces all the network traffic through that adapter. I don't know enough, but with auto metric enabled, traffic might not always go through the correct bridge interface on WSL.

svdHero · 2022-02-07T09:02:40Z

@rochecompaan @RandolphBack The fix is change MTU to 1350 using cmd netsh interface set ipv4 subinterface "vEthernet (WSL)" mtu=1350 store=persistent

and then do this: #4901 (comment)

@Equation-Group
When I paste that command into a CMD, I get the error message

The following command was not found: interface set ipv4 subinterface "vEthernet (WSL)" mtu=1350 store=persistent.

What's wrong here? I can enter netsh in order to get to the REPL just fine.

eqn-group · 2022-02-07T19:30:48Z

@svdHero there was a typo in the command, i corrected it.

jetnet · 2022-06-23T19:21:29Z

what helped a little in my case (client_loop: send disconnect: Broken pipe) to keep a connection longer is to set the ServerAliveCountMax parameter to a higher value, e.g. 100 in .ssh/config:

Host *
    ForwardAgent yes
    AddressFamily inet
    TCPKeepAlive yes
    ServerAliveInterval 60
    ServerAliveCountMax 100
    IPQoS 0

The following did not help, but currently in place:
in WSL:

# echo 1 | sudo tee /proc/sys/net/ipv4/tcp_mtu_probing # did not help, commented out
sudo ifconfig eth0 mtu 1350

in Windows:

netsh interface ipv4 set subinterface "vEthernet (WSL)" mtu=1350 store=persistent

I have not changed the interface metric settings, as I use an Ethernet adapter on my working station.

This is definitely a WSL2 issue, since the same .ssh/config is used for WSL ssh and for the Windows OpenSSH client.

I don't understand, why the ticket has been closed.
Guys at MS, you cannot imaging how much time I waste on working around this issue.

ghost · 2022-07-08T23:30:08Z

Using a proxy might help

xanderificnl · 2022-10-05T03:57:44Z

This is issue only started for me after enabling stricter cryptographic settings on the server. Connecting over the WireGuard tunnel went fine, straight via PowerShell too. Connecting to the servers' public IP didn't.

Changing the MTU did resolve the issue. I don't use a local ssh config for the client.

FWIW, it was coupled with a "sshd[1186]: fatal: mm_answer_sign: sign: error in libcrypto" on the server side.

jetnet · 2022-10-05T05:22:38Z

finally figured out: group policy update forces restart of WSL network interface: https://learn.microsoft.com/en-us/answers/questions/723272/group-policy-update-and-hyper-v-vethernet-disconne.html

devloloper · 2022-10-05T07:53:14Z

finally figured out: group policy update forces restart of WSL network interface: https://learn.microsoft.com/en-us/answers/questions/723272/group-policy-update-and-hyper-v-vethernet-disconne.html

Can confirm this, running "gpupdate /force" always leads to a ssh disconnect.

jbowler · 2023-02-18T04:26:38Z

Just a note: it's nothing to do with WSL. I can repro this on a Linux (gentoo) system with an ssh into localhost.

Clearly that is a bug, but my best guess is that it is caused by having a local system that is IPv6; if the IPv6 connection drops the local IPv6's change and I suspect that reveals a bug in ssh.

OneBlue added the needs-author-feedback label Jan 25, 2022

ghost removed the needs-author-feedback label Jan 26, 2022

rochecompaan closed this as completed Jan 31, 2022

norswap mentioned this issue Mar 4, 2022

Fix timeout in fault injection challenge scenario ethereum-optimism/cannon#53

Closed

shreyasmiraj mentioned this issue Nov 14, 2023

ssh connection sends client_loop: send disconnect: Broken pipe shreyasmiraj/CFsshTunnel#1

Open

github-actions bot mentioned this issue Jul 9, 2024

WSL network crashes after 100 ssh connections #11759

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSH to remote causes disconnect with client_loop: send disconnect: Broken pipe #7966

SSH to remote causes disconnect with client_loop: send disconnect: Broken pipe #7966

rochecompaan commented Jan 24, 2022 •

edited

Loading

eqn-group commented Jan 25, 2022 •

edited

Loading

rochecompaan commented Jan 25, 2022

rochecompaan commented Jan 25, 2022

rochecompaan commented Jan 26, 2022

rochecompaan commented Jan 26, 2022

eqn-group commented Jan 26, 2022 •

edited

Loading

rochecompaan commented Jan 26, 2022

victorpablosceruelo commented Jan 26, 2022 •

edited

Loading

rochecompaan commented Jan 26, 2022

RandolphBack commented Jan 27, 2022

eqn-group commented Jan 30, 2022 •

edited

Loading

rochecompaan commented Jan 31, 2022

rochecompaan commented Jan 31, 2022

svdHero commented Feb 7, 2022

eqn-group commented Feb 7, 2022

jetnet commented Jun 23, 2022 •

edited

Loading

ghost commented Jul 8, 2022

xanderificnl commented Oct 5, 2022 •

edited

Loading

jetnet commented Oct 5, 2022

devloloper commented Oct 5, 2022

jbowler commented Feb 18, 2023

SSH to remote causes disconnect with client_loop: send disconnect: Broken pipe #7966

SSH to remote causes disconnect with client_loop: send disconnect: Broken pipe #7966

Comments

rochecompaan commented Jan 24, 2022 • edited Loading

Version

WSL Version

Kernel Version

Distro Version

Other Software

Repro Steps

Expected Behavior

Actual Behavior

Diagnostic Logs

eqn-group commented Jan 25, 2022 • edited Loading

rochecompaan commented Jan 25, 2022

rochecompaan commented Jan 25, 2022

rochecompaan commented Jan 26, 2022

rochecompaan commented Jan 26, 2022

eqn-group commented Jan 26, 2022 • edited Loading

rochecompaan commented Jan 26, 2022

victorpablosceruelo commented Jan 26, 2022 • edited Loading

rochecompaan commented Jan 26, 2022

RandolphBack commented Jan 27, 2022

eqn-group commented Jan 30, 2022 • edited Loading

rochecompaan commented Jan 31, 2022

rochecompaan commented Jan 31, 2022

svdHero commented Feb 7, 2022

eqn-group commented Feb 7, 2022

jetnet commented Jun 23, 2022 • edited Loading

ghost commented Jul 8, 2022

xanderificnl commented Oct 5, 2022 • edited Loading

jetnet commented Oct 5, 2022

devloloper commented Oct 5, 2022

jbowler commented Feb 18, 2023

rochecompaan commented Jan 24, 2022 •

edited

Loading

eqn-group commented Jan 25, 2022 •

edited

Loading

eqn-group commented Jan 26, 2022 •

edited

Loading

victorpablosceruelo commented Jan 26, 2022 •

edited

Loading

eqn-group commented Jan 30, 2022 •

edited

Loading

jetnet commented Jun 23, 2022 •

edited

Loading

xanderificnl commented Oct 5, 2022 •

edited

Loading