crypto/tls: make maxHandshake larger or configurable

[awnumar]

What version of Go are you using (go version)?

$ go version
go version go1.17.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/awnumar/Library/Caches/go-build"
GOENV="/Users/awnumar/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/awnumar/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/awnumar"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.17.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.17.6/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s2/fftr5j0n443f4z2sm11wlxv40000gn/T/go-build2260917661=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

1. Generate a very large certificate. I passed mkcert many short SANs to get this.
2. Start a Go TLS server.
3. Connect to it with a Go TLS client.

What did you expect to see?

I expected the connection to succeed.

What did you see instead?

tls: handshake message of length 110013 bytes exceeds maximum of 65536 bytes

The limit is defined here:

const maxHandshake = 65536 // maximum handshake we support (protocol max is 16 MB)

and it says that the protocol max is 16 MB, not 64 KiB. I assume this hard limit is to mitigate resource exhaustion attacks, but it would be preferable if it was configurable.

Context

We dynamically generate certificates for mTLS within and across clusters and configure which services are allowed to open connections between each other using Subject Alternative Names. In extreme cases a service that needs to communicate with many other services could have a certificate that's larger than this limit.

This isn't a problem right now but a limit of 64 KiB provides a fairly small and uncomfortable safety buffer. Is there a reason that this limit is not configurable?

This issue is related to #35153 where @FiloSottile says "Documenting something makes it a backwards compatibility promise". Is making it configurable still possible while preserving backwards compatibility in the future?

[bcmills]

Is making it configurable still possible while preserving backwards compatibility in the future?

Yes: making something configurable is backwards-compatible as long as the default behavior remains the same. (We can be certain that no existing program configures it to anything other than the default.)

[seankhliao]

cc @golang/security