New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sshd rejects all connections with a broken pipe error #227

Closed
alexchandel opened this Issue May 23, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@alexchandel

alexchandel commented May 23, 2016

Attempting to connect as any user, after a password challenge and correct password entry, yields the error message

packet_write_wait: Connection to 192.168.1.100: Broken pipe

where 192.168.1.100 is the server's IP address.

This happens in the 2016.05.15 build, with key-based authentication enabled, even though it's not being used here.

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel May 23, 2016

This also occurs when installing it without key-based authentication. sshd seems to be completely broken in this release.

alexchandel commented May 23, 2016

This also occurs when installing it without key-based authentication. sshd seems to be completely broken in this release.

@manojampalam

This comment has been minimized.

Show comment
Hide comment
@manojampalam

manojampalam May 23, 2016

Collaborator

sshd certainly is not completely broken. All the steps detailed in wiki are tested and validated before release.
See if this can help: https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting%20Steps

Collaborator

manojampalam commented May 23, 2016

sshd certainly is not completely broken. All the steps detailed in wiki are tested and validated before release.
See if this can help: https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting%20Steps

@silarsis

This comment has been minimized.

Show comment
Hide comment
@silarsis

silarsis May 24, 2016

I got the same error. Running sshd with '-d' I get:

debug1: userauth-request for user kevin service ssh-connection method password
debug1: attempt 2 failures 1
Accepted password for kevin from 10.1.1.21 port 55781 ssh2
debug1: Entering interactive session for SSH2.
debug1: pipe - read end: handle:00000000000001C0, io:0000021595656800, fd:4
debug1: pipe - write end: handle:0000000000000208, io:0000021595656A10, fd:6
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc console
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on console for kevin from 10.1.1.21 port 55781
debug1: pipe - read end: handle:000000000000020C, io:0000021595656330, fd:7
debug1: pipe - write end: handle:0000000000000210, io:0000021595656960, fd:8
debug1: pipe - read end: handle:0000000000000224, io:00000215956565F0, fd:9
debug1: pipe - write end: handle:0000000000000228, io:00000215956563E0, fd:10
debug1: pipe - read end: handle:000000000000022C, io:0000021595656490, fd:11
debug1: pipe - write end: handle:0000000000000230, io:0000021595656540, fd:12
debug1: -> GetHomeDirFromToken()...
debug1: <- GetHomeDirFromToken()...
debug1: Executing command: ssh-shellhost.exe
debug1: ERROR. Cannot create process as new user (1314).

Client-side I have:

debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: recv - from CB ERROR:10054, io:0000029C991455F0
debug1: send - WSASend() ERROR:54, io:0000029C991455F0
debug1: send - WSASend() ERROR:54, io:0000029C991455F0
packet_write_wait: Connection to 10.1.1.27: Unknown error
C:\Program Files\OpenSSH-Win64>

I'm not sure whether this is the actual issue causing the broken pipe, or whether this is #207 masking the other issue.

silarsis commented May 24, 2016

I got the same error. Running sshd with '-d' I get:

debug1: userauth-request for user kevin service ssh-connection method password
debug1: attempt 2 failures 1
Accepted password for kevin from 10.1.1.21 port 55781 ssh2
debug1: Entering interactive session for SSH2.
debug1: pipe - read end: handle:00000000000001C0, io:0000021595656800, fd:4
debug1: pipe - write end: handle:0000000000000208, io:0000021595656A10, fd:6
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc console
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on console for kevin from 10.1.1.21 port 55781
debug1: pipe - read end: handle:000000000000020C, io:0000021595656330, fd:7
debug1: pipe - write end: handle:0000000000000210, io:0000021595656960, fd:8
debug1: pipe - read end: handle:0000000000000224, io:00000215956565F0, fd:9
debug1: pipe - write end: handle:0000000000000228, io:00000215956563E0, fd:10
debug1: pipe - read end: handle:000000000000022C, io:0000021595656490, fd:11
debug1: pipe - write end: handle:0000000000000230, io:0000021595656540, fd:12
debug1: -> GetHomeDirFromToken()...
debug1: <- GetHomeDirFromToken()...
debug1: Executing command: ssh-shellhost.exe
debug1: ERROR. Cannot create process as new user (1314).

Client-side I have:

debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: recv - from CB ERROR:10054, io:0000029C991455F0
debug1: send - WSASend() ERROR:54, io:0000029C991455F0
debug1: send - WSASend() ERROR:54, io:0000029C991455F0
packet_write_wait: Connection to 10.1.1.27: Unknown error
C:\Program Files\OpenSSH-Win64>

I'm not sure whether this is the actual issue causing the broken pipe, or whether this is #207 masking the other issue.

@manojampalam

This comment has been minimized.

Show comment
Hide comment
@manojampalam

manojampalam May 24, 2016

Collaborator

yes this is #207. Try doing this with included ntrights.exe

ntrights.exe -u +r SeAssignPrimaryTokenPrivilege

and log off, log in.

Collaborator

manojampalam commented May 24, 2016

yes this is #207. Try doing this with included ntrights.exe

ntrights.exe -u +r SeAssignPrimaryTokenPrivilege

and log off, log in.

@silarsis

This comment has been minimized.

Show comment
Hide comment
@silarsis

silarsis May 24, 2016

When running sshd.exe -d after executing that ntrights command and logging out/logging back in, it all works perfectly - connection is accepted and I'm logged in. Although I get periodic "select - timing out" messages on the client side that seem to lock things up - the server shows "keepalive@openssh.com want_reply 1" messages around the same time.

When I exit the debug, and run net start sshd then try to ssh in, I'm back to it failing with the packer_write_wait error on the client side after providing my password.

silarsis commented May 24, 2016

When running sshd.exe -d after executing that ntrights command and logging out/logging back in, it all works perfectly - connection is accepted and I'm logged in. Although I get periodic "select - timing out" messages on the client side that seem to lock things up - the server shows "keepalive@openssh.com want_reply 1" messages around the same time.

When I exit the debug, and run net start sshd then try to ssh in, I'm back to it failing with the packer_write_wait error on the client side after providing my password.

@manojampalam

This comment has been minimized.

Show comment
Hide comment
@manojampalam

manojampalam May 24, 2016

Collaborator

hmm. can you try this and restart sshd?
ntrights.exe -u "NT SERVICE\SSHD" +r SeAssignPrimaryTokenPrivilege

Collaborator

manojampalam commented May 24, 2016

hmm. can you try this and restart sshd?
ntrights.exe -u "NT SERVICE\SSHD" +r SeAssignPrimaryTokenPrivilege

@silarsis

This comment has been minimized.

Show comment
Hide comment
@silarsis

silarsis May 24, 2016

Yup, that's done the trick.

I'm on Win10 Version 1511 Build 14342.1001, fwiw - latest insiders build afaik.

It dropped me in C:\Windows when I logged in, too - not my home dir.

silarsis commented May 24, 2016

Yup, that's done the trick.

I'm on Win10 Version 1511 Build 14342.1001, fwiw - latest insiders build afaik.

It dropped me in C:\Windows when I logged in, too - not my home dir.

@manojampalam

This comment has been minimized.

Show comment
Hide comment
@manojampalam

manojampalam May 24, 2016

Collaborator

See release notes (c:\windows is a known issue that I'm working on)

install-sshd.ps1 should have run ntrights.exe command. Was that not run?

Collaborator

manojampalam commented May 24, 2016

See release notes (c:\windows is a known issue that I'm working on)

install-sshd.ps1 should have run ntrights.exe command. Was that not run?

@silarsis

This comment has been minimized.

Show comment
Hide comment
@silarsis

silarsis May 24, 2016

ssh-agent.log:

C:\Program Files\OpenSSH-Win64> cat .\ssh-agent.log
3136 13:07:22 788 agent_start pid:3136, dbg:0, child:0, pipe:0
3136 13:07:55 126 client pid 2724 connected on \\.\pipe\ssh-keyagent
3136 13:07:55 142 debug1: spawned child 11324 to process 3
11324 13:07:55 184 agent_start pid:11324, dbg:0, child:1, pipe:476
11324 13:07:55 184 debug1: connection io 000002751628B650 #bytes:0 state:0
11324 13:08:12 792 debug1: iocp error: 109 on 000002751628B650

11324 13:08:12 792 debug1: connection 000002751628B650 clean up
11324 13:08:12 792 debug1: iocp error: 6 on 0000000000000000

C:\Program Files\OpenSSH-Win64>

I'm not actually seeing an sshd.log - and I ran Get-ChildItem / -Recurse -Filter 'sshd.log'. That's with sshd_config LogLevel set to "DEBUG".

silarsis commented May 24, 2016

ssh-agent.log:

C:\Program Files\OpenSSH-Win64> cat .\ssh-agent.log
3136 13:07:22 788 agent_start pid:3136, dbg:0, child:0, pipe:0
3136 13:07:55 126 client pid 2724 connected on \\.\pipe\ssh-keyagent
3136 13:07:55 142 debug1: spawned child 11324 to process 3
11324 13:07:55 184 agent_start pid:11324, dbg:0, child:1, pipe:476
11324 13:07:55 184 debug1: connection io 000002751628B650 #bytes:0 state:0
11324 13:08:12 792 debug1: iocp error: 109 on 000002751628B650

11324 13:08:12 792 debug1: connection 000002751628B650 clean up
11324 13:08:12 792 debug1: iocp error: 6 on 0000000000000000

C:\Program Files\OpenSSH-Win64>

I'm not actually seeing an sshd.log - and I ran Get-ChildItem / -Recurse -Filter 'sshd.log'. That's with sshd_config LogLevel set to "DEBUG".

@joelnb

This comment has been minimized.

Show comment
Hide comment
@joelnb

joelnb May 29, 2016

I am also getting this issue when installing the chocolatey package with:

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
$env:Path += ";C:\\ProgramData\\Chocolatey\\bin"

choco install win32-openssh -params '"/SSHServerFeature /KeyBasedAuthenticationFeature"' -confirm

The suggested fix also worked for me but I get no logging even if I set LogLevel DEBUG in sshd_config (I didn't have the sshd.log at all but when I created it there was still no logging).

EDIT: Possibly related to this? But I assumed that was just setting the LogLevel QUIET which was in sshd_config originally.

WARNING: Explicitly disabling sshd logging as it currently logs about .5 GB / hour

joelnb commented May 29, 2016

I am also getting this issue when installing the chocolatey package with:

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
$env:Path += ";C:\\ProgramData\\Chocolatey\\bin"

choco install win32-openssh -params '"/SSHServerFeature /KeyBasedAuthenticationFeature"' -confirm

The suggested fix also worked for me but I get no logging even if I set LogLevel DEBUG in sshd_config (I didn't have the sshd.log at all but when I created it there was still no logging).

EDIT: Possibly related to this? But I assumed that was just setting the LogLevel QUIET which was in sshd_config originally.

WARNING: Explicitly disabling sshd logging as it currently logs about .5 GB / hour
@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Jun 1, 2016

Actually the ntrights powershell/.NET error stops you from installing at all when you're dealing with this. You have to restart the command prompt after trying to install the first time, so the path can be updated, then use the ntrights executable manually.

alexchandel commented Jun 1, 2016

Actually the ntrights powershell/.NET error stops you from installing at all when you're dealing with this. You have to restart the command prompt after trying to install the first time, so the path can be updated, then use the ntrights executable manually.

@manojampalam

This comment has been minimized.

Show comment
Hide comment
@manojampalam

manojampalam Jul 19, 2016

Collaborator

Reopen if needed

Collaborator

manojampalam commented Jul 19, 2016

Reopen if needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment