Skip to content

os,internal/poll: issues with overlapped I/O #77142

@idcmp

Description

@idcmp

Go version

go version 1.25.5 darwin/64

Note: I built off of release-branch.go1.25 with the latest ref of 525dd8.

I'm cross-compiling with GOOS=windows GOARCH=amd64.

Output of go env in your module/workspace:

Unable due to work security. I may be able to offer specific values if needed.

What did you do?

Background

ssh supports configuring a "proxycommand" in ~/.ssh/config. This proxycommand is expected to make the TCP connection to the remote host and offer it up over stdin and stdout. You could imagine nc (netcat) as the world's simplest proxycommand.

In our situation, we have a Go binary that acts as a proxycommand. It does a bunch of heavy lifting and ultimately honours the contract of proxycommand. For the sake of discussion, we'll call this the "Simple Forwarding Tool" (sft).

With the combination of a Match stanza in ~/.ssh/config, an end user can type: ssh somewhere and OpenSSH will automatically execute the sft proxycommand %h:

Match exec "sft resolve -q %h"
   ProxyCommand sft proxycommand %h

Similarly, a user can type scp filename.txt somewhere:/tmp/ to perform an scp proxied by the proxycommand.

Situation

In https://go-review.googlesource.com/c/go/+/664415 a change was introduced that now breaks sft's ability to proxy scp when running on Windows. I've tested this by rebuilding multiple times with git bisect and it has identified the ref of this CL as the culprit.

$ ssh somewhere <- works fine
$ scp something.txt somewhere:/tmp/ <- does not work

What did you see happen?

I've broken the logs up here into two parts: setup, and error. I'm not sure if the setup is interesting, but it's included for completeness. Setup starts with the Simple Forwarding Tool (sft) offering itself up as a proxycommand by announcing itself as SSH-2.0-SFT-PROXY2...

Setup

debug1: Remote protocol version 2.0, remote software version SFT-PROXY2
debug1: compat_banner: no match: SFT-PROXY2
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to ip-172-31-0-95:22 as 'administrator'
debug3: record_hostkey: found key type RSA in file C:\\Users\\ADMINI~1\\AppData\\Local\\ScaleFT\\proxycommand_known_hosts:3
debug3: load_hostkeys_file: loaded 1 keys from ip-172-31-0-95
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: prefer hostkeyalgs: [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com),[rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com),rsa-sha2-512,rsa-sha2-256
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org),ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,[kex-strict-c-v00@openssh.com](mailto:kex-strict-c-v00@openssh.com)
debug2: host key algorithms: [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com),[rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com),rsa-sha2-512,rsa-sha2-256,[ssh-ed25519-cert-v01@openssh.com](mailto:ssh-ed25519-cert-v01@openssh.com),[ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp256-cert-v01@openssh.com),[ecdsa-sha2-nistp384-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp384-cert-v01@openssh.com),[ecdsa-sha2-nistp521-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp521-cert-v01@openssh.com),[sk-ssh-ed25519-cert-v01@openssh.com](mailto:sk-ssh-ed25519-cert-v01@openssh.com),[sk-ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:sk-ecdsa-sha2-nistp256-cert-v01@openssh.com),ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com),[sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com)
debug2: ciphers ctos: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com),aes128-ctr,aes192-ctr,aes256-ctr,[aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com),[aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com)
debug2: ciphers stoc: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com),aes128-ctr,aes192-ctr,aes256-ctr,[aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com),[aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com)
debug2: MACs ctos: [umac-64-etm@openssh.com](mailto:umac-64-etm@openssh.com),[umac-128-etm@openssh.com](mailto:umac-128-etm@openssh.com),[hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com),[hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com),[umac-64@openssh.com](mailto:umac-64@openssh.com),[umac-128@openssh.com](mailto:umac-128@openssh.com),hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: [umac-64-etm@openssh.com](mailto:umac-64-etm@openssh.com),[umac-128-etm@openssh.com](mailto:umac-128-etm@openssh.com),[hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com),[hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com),[umac-64@openssh.com](mailto:umac-64@openssh.com),[umac-128@openssh.com](mailto:umac-128@openssh.com),hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none,[zlib@openssh.com](mailto:zlib@openssh.com),zlib
debug2: compression stoc: none,[zlib@openssh.com](mailto:zlib@openssh.com),zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org),ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[kex-strict-s-v00@openssh.com](mailto:kex-strict-s-v00@openssh.com)
debug2: host key algorithms: rsa-sha2-256,rsa-sha2-512,ssh-rsa
debug2: ciphers ctos: [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com),[aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com),[chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com),aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com),[aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com),[chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com),aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com),[hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com),hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com),[hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com),hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) MAC: <implicit> compression: none
debug1: kex: client->server cipher: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:hAlx+JWT6pe+xV9h/hk7N+lGMmk29sW6vydWveD6t4g
debug3: record_hostkey: found key type RSA in file C:\\Users\\ADMINI~1\\AppData\\Local\\ScaleFT\\proxycommand_known_hosts:3
debug3: load_hostkeys_file: loaded 1 keys from ip-172-31-0-95
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ip-172-31-0-95' is known and matches the RSA host key.
debug1: Found key in C:\\Users\\ADMINI~1\\AppData\\Local\\ScaleFT\\proxycommand_known_hosts:3
debug3: send packet: type 21
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug3: ssh_get_authentication_socket_path: path '\\\\.\\pipe\\openssh-ssh-agent'
debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2
debug1: get_agent_identities: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_ecdsa_sk
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_ed25519_sk
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_xmss
debug1: Will attempt key: C:\\Users\\Administrator/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com),[sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com),ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: [ping@openssh.com](mailto:ping@openssh.com)=<0>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
2026-01-09T23:48:33.334Z        INFO    bridge: Server started
debug3: receive packet: type 52
debug3: w32_getpeername ERROR: not sock :2
Authenticated to ip-172-31-0-95 (via proxy) using "none".
debug2: fd 3 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: proc
debug3: client_repledge: enter
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug3: w32_getpeername ERROR: not sock :2
debug2: client_session2_setup: id 0
debug1: Sending subsystem: sftp
debug2: channel 0: request subsystem confirm 1
debug3: send packet: type 98
debug3: client_repledge: enter
debug1: pledge: fork
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
2026-01-09T23:48:33.340Z        DEBUG   Session Request {"request-type": "subsystem"}
2026-01-09T23:48:33.340Z        DEBUG   Session starting request        {"request-type": "subsystem"}
2026-01-09T23:48:33.851Z        DEBUG   Received Request        {"type": "[hostkeys-00@openssh.com](mailto:hostkeys-00@openssh.com)", "want_reply": false}
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [hostkeys-00@openssh.com](mailto:hostkeys-00@openssh.com) want_reply 0
2026-01-09T23:48:33.876Z        DEBUG   Not forwarding client trust: IsForwardClientTrustEnabled is false
2026-01-09T23:48:33.878Z        INFO    RecordSpan      {"start": "2026-01-09T23:48:33.876Z", "duration": "2.0104ms", "traceID": 4188539475582866215, "spanID": 3198427524659839269, "t": "trace", "tags": null, "operation": "proxy.client.ForwardClientTrust"}
2026-01-09T23:48:33.879Z        INFO    RecordSpan      {"tags": null, "operation": "proxy.client.ForwardAgent", "start": "2026-01-09T23:48:33.879Z", "duration": "0s", "traceID": 4188539475582866215, "t": "trace", "spanID": 3606750605364129388}
2026-01-09T23:48:33.880Z        DEBUG   Channel request status  {"ok": true, "request_type": "subsystem", "req.want_reply": true, "component": "proxiedChannel"}
2026-01-09T23:48:33.881Z        DEBUG   Opening Channel {"channel_id": 1, "type": "session"}

Error

debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: subsystem request accepted on channel 0
debug3: write ERROR from cb(2):87, io:0000018EE34AACD0
debug2: channel 0: write failed
debug2: chan_shutdown_write: channel 0: (i0 o0 sock -1 wfd 6 efd 7 [write])
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
2026-01-09T23:49:12.456Z        DEBUG   bridge: Context is done, shutting down.
2026-01-09T23:49:12.456Z        DEBUG   proxyIO: done   {"pdid": 1, "direction": "client->server", "name": "channel"}
2026-01-09T23:49:12.457Z        DEBUG   closingInOrder: Waiting on exit-status  {"pdid": 1, "direction": "client->server"}
2026-01-09T23:49:12.456Z        DEBUG   proxyIO: done   {"name": "stderr", "pdid": 1, "direction": "client->server"}
2026-01-09T23:49:12.456Z        DEBUG   proxyReqs: done {"pdid": 1, "direction": "client->server"}
2026-01-09T23:49:12.458Z        DEBUG   closingInOrder: got exit-status via close       {"pdid": 1, "direction": "client->server"}
2026-01-09T23:49:12.456Z        DEBUG   proxyIO: done   {"direction": "server->client", "pdid": 1, "name": "stderr"}
2026-01-09T23:49:12.456Z        DEBUG   proxyIO: done   {"pdid": 1, "direction": "server->client", "name": "channel"}
2026-01-09T23:49:12.456Z        DEBUG   proxyReqs: done {"direction": "server->client", "pdid": 1}
2026-01-09T23:49:12.456Z        DEBUG   global requests: Context is done, shutting down.
2026-01-09T23:49:12.458Z        DEBUG   closingInOrder: half-closing dst        {"direction": "client->server", "pdid": 1}
2026-01-09T23:49:12.458Z        DEBUG   closingInOrder: Waiting on exit-status  {"pdid": 1, "direction": "server->client"}
2026-01-09T23:49:12.459Z        DEBUG   Proxy: done     {"pdid": 1, "direction": "client->server"}
2026-01-09T23:49:12.459Z        DEBUG   closingInOrder: got exit-status via close       {"direction": "server->client", "pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   proxy2.Run: pdA done    {"pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   closingInOrder: half-closing dst        {"direction": "server->client", "pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   Proxy: done     {"direction": "server->client", "pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   proxy2.Run: pdB done    {"pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   proxy2.Run: wait over   {"pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   proxy2.Run: done        {"pdid": 1}
2026-01-09T23:49:12.459Z        DEBUG   Channel Closed  {"channel_id": 1, "type": "session"}
2026-01-09T23:49:12.459Z        INFO    RecordSpan      {"traceID": 4188539475582866215, "spanID": 3510809545581562661, "tags": {"error":false}, "operation": "cli.proxycommand", "start": "2026-01-09T23:48:32.479Z", "t": "trace", "duration": "39.9804923s"}
C:\Windows\System32\OpenSSH\scp.exe: Connection closed
debug3: send packet: type 1
debug3: write ERROR from cb(2):232, io:0000018EE34AD0C0
client_loop: send disconnect: Unknown error

No file is transmitted to the host.

What did you expect to see?

I mean, I expected an scp to happen and a file to appear. :-)

Metadata

Metadata

Assignees

Labels

NeedsDecisionFeedback is required from experts, contributors, and/or the community before a change can be made.OS-Windowscompiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type
No fields configured for issues without a type.

Projects

Status

Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions