crypto/tls: slow(er) creation of TLS/SSL clients under certain environments

[hallm4]

Go version

go version go1.24.4 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHEPROG=''
GODEBUG=''
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3148247925=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/go/src/go.mod'
GOOS='linux'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'

GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.4'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Under certain OS's, a simple TLS client will incur a large(for various definitions of large of which I will get to) amount of extra time as part of the connection creation.

Take the following snippet :

func tlsdial() {

	start := time.Now()
	conn, err := tls.Dial("tcp", "<pickALocalSSLHost>:443", &tls.Config{})
	if err != nil {
		log.Fatalf("failed to tls.Dial host: %v", err)
	}
	conn.Close()
	fmt.Printf("Call tls dial #%v: error - %v, duration - %v\n", callCounter, nil, time.Since(start))

}

What did you see happen?

If we create a go binary, and run this a few times under amazonlinux:2, we'll see a pretty quick response.

If we run this a few times under amazonlinux:2023, we'll see some extra time (1-2s). Why?

Turns out, there are a few other things at play here, namely SSL_CERT_FILE and importantly, SSL_CERT_DIR.

Under amazonlinux:2 the /etc/ssl/certs dir is a symlink, to a mostly empty location.

Under amazonlinux:2023 the /etc/ssl/certs dir is a symlink to a directory with over ~740 PEM files.

Go will then automatically load every PEM file it sees, greedily.

https://github.com/golang/go/blob/master/src/crypto/x509/root_unix.go

go/src/crypto/x509/root_linux.go

Lines 9 to 17 in 53af292

	// Possible certificate files; stop after finding one.
	var certFiles = []string{
	"/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc.
	"/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 6
	"/etc/ssl/ca-bundle.pem", // OpenSUSE
	"/etc/pki/tls/cacert.pem", // OpenELEC
	"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7
	"/etc/ssl/cert.pem", // Alpine Linux
	}

go/src/crypto/x509/root_linux.go

Lines 20 to 22 in 53af292

	var certDirectories = []string{
	"/etc/ssl/certs", // SLES10/SLES11, https://golang.org/issue/12139
	"/etc/pki/tls/certs", // Fedora/RHEL

When you are creating SSL clients / DynamoDB clients under AWS Lambda, this extra time is actually noticeable.

What did you expect to see?

Connection times should be near the same.

Would hope there would be a good way to change this behaviour, or turn SSL_CERT_FILE and SSL_CERT_DIR behaviour into a OR procedure, instead of AND. SSL_CERT_FILE exists in amazonlinux:2023 and has the full set of certificates already.

AWS via our support channels has acknowledged this extra load time, but do not wish to fix it in their images, as they are simply bringing in an amalgamation of Fedora, etc, as amazonlinux:2023.

Our only current workaround is either to completely roll our own tls.Config or hack the environment in our lambda terraform or Go.

      # Either to empty dir (/tmp) or to non-existent
      SSL_CERT_DIR          = "/fdgdfgdasdfsdfsd/dgfdf"

func main() {
        os.Setenv("SSL_CERT_DIR", "/this/does/not/exist")
        lambda.Start(Run)
}

[hallm4]

Kind of related to the discussion in #71924

[gabyhelp]

Related Issues

• crypto/tls: connection speed decreases significantly when enable TLS #22761 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

as I understand, this is a one time cost to load the system roots, which is cached for further reuse. I don't think we'd add any more knobs than we have now, if you want to skip a certain set of roots, setting the corresponding env to an empty location is the way to go.

[trivenay]

For anyone hitting this: PR #80374 implements a fix that defers the redundant directory scan when a cert bundle is available.

The profiling data shows the root cause is not symlink duplication but rather that Go eagerly reads ~253 individual cert files from /etc/ssl/certs/ after already loading the same certificates from the bundle file.

Profiling breakdown (128MB Lambda, 0.08 vCPU, Amazon Linux 2023):

Read bundle:        134ms (251 certs)
Directory scan:     820ms (253 files read, 0 new certs)

Results (first TLS call latency, 128MB Lambda):

Environment	Before	After	Improvement
Amazon Linux 2023	~1510ms	~398ms	-74%
Amazon Linux 2	~704ms	~395ms	-44%

No env var workaround needed.

[gopherbot]

Change https://go.dev/cl/799801 mentions this issue: crypto/x509: defer directory scan when cert bundle provides roots