x/crypto/ssh: SSH_MSG_USERAUTH_PK_OK handling differs from OpenSSH

[stephenm-stripe]

Go version

go version go1.20.5 darwin/arm64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/stephenm/Library/Caches/go-build"
GOENV="/Users/stephenm/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/stephenm/go/pkg/mod"
GONOPROXY="*.corp.stripe.com,github.com/stripe-internal/*,github.com/stripe/*"
GONOSUMDB="*.corp.stripe.com,github.com/stripe-internal/*,github.com/stripe/*"
GOOS="darwin"
GOPATH="/Users/stephenm/go"
GOPRIVATE="*.corp.stripe.com,github.com/stripe-internal/*,github.com/stripe/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.20.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/stephenm/stripe/crypto/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/nd/8x6xvrgs3fb1ngn08_rx59t40000gn/T/go-build4288886961=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I was encountering the same error in #54027 for connecting to a remote server outside of my control. In this scenario, the remote server does not follow the RFC in creating SSH_MSG_USERAUTH_PK_OK packets.

The RFC specifies that SSH_MSG_USERAUTH_REQUEST packets include string public key algorithm name and the server would respond with a SSH_MSG_USERAUTH_PK_OK that includes string public key algorithm name from the request.

In this case, I would see our SSH_MSG_USERAUTH_REQUEST include an algorithm like rsa-sha2-256, but the SSH_MSG_USERAUTH_PK_OK response would always contain ssh-rsa for all RSA algorithms provided.

What did you expect to see?

Go SSH handshakes have the same outcome as OpenSSH handshakes for the same parameters.

What did you see instead?

OpenSSH handshakes succeed with the remote SSH server while Go SSH handshakes do not.

The Go handshake fails because the handling here checks that the outbound algorithm string matches the one in the response. In our case, the remote server is not following the RFC and sending back the type rather than the algorithm string that was sent to it. So Go's implementation considers this a failed auth and returns an error to the application.

Yet, OpenSSH via ssh on the command line succeeds for this server. The OpenSSH implementation for handling SSH_MSG_USERAUTH_PK_OK doesn't check the exact string matches, rather, it converts the algorithm names into the key types and compares on that (code for the check, code for the struct).

I admit, OpenSSH's handling here differs from the RFC, but it does result in a successful connection while Go's implementation does not. I also tried commenting out Go's logic around key validation that invokes these packets and the connection succeeded, so the broken invariant doesn't actually impact the connection overall.

I will put up a pull request to modify Go's implementation of SSH_MSG_USERAUTH_PK_OK checks to match OpenSSH's.

[gopherbot]

Change https://go.dev/cl/550716 mentions this issue: x/crypto/ssh: update SSH_MSG_USERAUTH_PK_OK handling to match OpenSSH

[thanm]

Thanks for the report. You mention that you are using Go 1.20 -- I assume the same problem exists in 1.21 and on tip?

@drakkan @golang/security per owners.

[drakkan]

Hello @stephenm-stripe, thanks for the detailed info. So this is yet another egde case for RSA keys.

Can you please confirm that the following patch works with your server? (also for certificates)

0001-ssh-relax-algo-check-in-SSH_MSG_USERAUTH_PK_OK.zip

[stephenm-stripe]

@thanm I haven't tested it on 1.21 or tip, but since this is in x/crypto, I don't think those will influence the outcome unless I misunderstand?

@drakkan Thanks for the patch. I have tested it against the non-compliant server and confirm it works.

[drakkan]

@thanm I haven't tested it on 1.21 or tip, but since this is in x/crypto, I don't think those will influence the outcome unless I misunderstand?

@drakkan Thanks for the patch. I have tested it against the non-compliant server and confirm it works.

Thanks for confirming. Can you please modify your CL like my proof of concept? I think it is more readable to have separate conditions for each case

[gopherbot]

Change https://go.dev/cl/553035 mentions this issue: x/crypto/ssh: update SSH_MSG_USERAUTH_PK_OK handling to match OpenSSH

[stephenm-stripe]

Thanks for confirming. Can you please modify your CL like my proof of concept? I think it is more readable to have separate conditions for each case

Yes. I meant to update the existing one, but have created https://go-review.googlesource.com/c/crypto/+/553035 instead.

[FiloSottile]

I'd like to push back a little on this.

The rsa-sha2-256 / ssh-rsa transition is already extremely complex, and quirk compatibility edge cases add complexity that might invalidate safety assumptions at a distance. They are also a maintenance burned going forward.

We do add quirk workarounds when they are necessary for ecosystem reasons (i.e. if five OpenSSH versions in a row exhibited a broken behavior), but only after there was an attempt to fix the issue on the peer's side, and only if the broken peer is popular.

a remote server outside of my control

Can you share more about what implementation has this issue? Is there a fix? How common is it?

[Melinysh]

Replying from my personal account (as mentioned in this comment).

a remote server outside of my control

Can you share more about what implementation has this issue? Is there a fix? How common is it?

I don't have details on the implementation, other than it was a third-party enterprise software suite for Windows. There is no known fix. We did inform the server owners of this limitation of the vendor software they are using, but they seemed unlikely to pursue alternative solutions given that we were their only client experiencing SSH issues (I suspect we were the only ones using Go to connect). On our side, of the many servers we connected to for SFTP purposes, this was the only remote server that had this issue.

[gopherbot]

Change https://go.dev/cl/573360 mentions this issue: ssh: validate key type in SSH_MSG_USERAUTH_PK_OK response