crypto/tls: message too long for RSA public key size

[bradfitz]

The github.com/bradfitz/http2 curl interop tests (which require Docker; see the Makefile) stopped working as of SHA-384 signatures in 1c10598

In this test, curl running in a Docker container connects to Go over TLS on localhost. I don't know what TLS configuration curl is trying to use.

ante:http2 $ go test -a -v -run=Lenient
=== RUN TestServerWithCurl_LenientCipherSuites
2015/05/06 12:53:23 http: TLS handshake error from 127.0.0.1:37678: failed to sign ECDHE parameters: crypto/rsa: message too long for RSA public key size
--- FAIL: TestServerWithCurl_LenientCipherSuites (0.38s)
        server_test.go:2153: Running test server for curl to hit at: https://127.0.0.1:42582
        server_test.go:2168: exit status 35: * Rebuilt URL to: https://127.0.0.1:42582/
                *   Trying 127.0.0.1...
                * Connected to 127.0.0.1 (127.0.0.1) port 42582 (#0)
                * successfully set certificate verify locations:
                *   CAfile: /etc/ssl/certs/ca-certificates.crt
                  CApath: none
                * TLSv1.2, TLS handshake, Client hello (1):
                } [272 bytes data]
                * TLSv1.2, TLS handshake, Server hello (2):
                { [62 bytes data]
                * NPN, negotiated HTTP2 (h2-14)
                * TLSv1.2, TLS handshake, CERT (11):
                { [389 bytes data]
                * TLSv1.2, TLS alert, Server hello (2):
                { [2 bytes data]
                * error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
                * Closing connection 0

[agl]

Is this still happening. I tried to get Docker running but failed ("There are no more loopback devices available."). I would assume that this is caused because the test keys are 512-bit, but the keys in the source appear to be 2048 and I don't see any problematically short ones.

[rsc]

Brad, is this still happening? Regardless, the bug is likely in crypto/tls not crypto/rsa.

[bradfitz]

Yes, this is still happening at tip, and is a regression from Go 1.4.

ante:http2 $ go test
2015/06/29 09:53:18 http: TLS handshake error from 127.0.0.1:58146: failed to sign ECDHE parameters: crypto/rsa: message too long for RSA public key size
--- FAIL: TestServerWithCurl (0.58s)
        server_test.go:2153: Running test server for curl to hit at: https://127.0.0.1:39155
        server_test.go:2168: exit status 35: * Rebuilt URL to: https://127.0.0.1:39155/
                *   Trying 127.0.0.1...
                * Connected to 127.0.0.1 (127.0.0.1) port 39155 (#0)
                * successfully set certificate verify locations:
                *   CAfile: /etc/ssl/certs/ca-certificates.crt
                  CApath: none
                * TLSv1.2, TLS handshake, Client hello (1):
                } [272 bytes data]
                * TLSv1.2, TLS handshake, Server hello (2):
                { [62 bytes data]
                * NPN, negotiated HTTP2 (h2-14)
                * TLSv1.2, TLS handshake, CERT (11):
                { [389 bytes data]
                * TLSv1.2, TLS alert, Server hello (2):
                { [2 bytes data]
                * error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
                * Closing connection 0

2015/06/29 09:53:18 http: TLS handshake error from 127.0.0.1:55516: failed to sign ECDHE parameters: crypto/rsa: message too long for RSA public key size
--- FAIL: TestServerWithCurl_LenientCipherSuites (0.53s)
        server_test.go:2153: Running test server for curl to hit at: https://127.0.0.1:36874
        server_test.go:2168: exit status 35: * Rebuilt URL to: https://127.0.0.1:36874/
                *   Trying 127.0.0.1...
                * Connected to 127.0.0.1 (127.0.0.1) port 36874 (#0)
                * successfully set certificate verify locations:
                *   CAfile: /etc/ssl/certs/ca-certificates.crt
                  CApath: none
                * TLSv1.2, TLS handshake, Client hello (1):
                } [272 bytes data]
                * TLSv1.2, TLS handshake, Server hello (2):
                { [62 bytes data]
                * NPN, negotiated HTTP2 (h2-14)
                * TLSv1.2, TLS handshake, CERT (11):
                { [389 bytes data]
                * TLSv1.2, TLS alert, Server hello (2):
                { [2 bytes data]
                * error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
                * Closing connection 0

FAIL
exit status 1
FAIL    github.com/bradfitz/http2       2.793s
ante:http2 $ go version
go version devel +434e0bc Mon Jun 29 16:07:14 2015 +0000 linux/amd64

[ebfe]

The negotiated cipher suite is TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384.

Signing fails because the message to sign (pkcs overhead + hash) are now > 512 bits and a 512-bit RSA key is used.

The problem disappears if the test cert/key in net/http/httptest/server.go are
replaced with 1024-bit versions.

[bradfitz]

Thanks @ebfe. Okay, I can update httptest.

[gopherbot]

CL https://golang.org/cl/11720 mentions this issue.