crypto/x509: incompatible behavior of SSL_CERT_DIR with OpenSSL

[DanielShaulov]

Go version

go version go1.22.2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dani/.cache/go-build'
GOENV='/home/dani/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dani/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dani/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.22'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.22/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1471836164=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Running with SSL_CERT_DIR directed to a location with a file named root.ca with root certificates

What did you see happen?

Golang loaded the root certs from the file and used them to verify SSL connections

What did you expect to see?

The SSL_CERT_DIR is only used in OpenSSL to look for files with a single certificate inside and the name should be the hash of the certificate.
The process is documented here.
The documentation above the const in the code even links to the man page for c_rehash tool that goes over a directory and creates those hashed files (links) - so that OpenSSL can consume them.

In this comment someone validated that this env var is being treated as a comma separated list - but that function is only called from X509_LOOKUP_hash_dir, which uses that hash technique from above.

[cagedmantis]

@FiloSottile @rolandshoemaker @cpu @golang/security

[seankhliao]

Is it really incompatible? It sounds like go just allows more files than openssl, but if you use openssl file naming everything still works

[DanielShaulov]

Is it really incompatible? It sounds like go just allows more files than openssl, but if you use openssl file naming everything still works

Sorry for the delayed response, missed the message.
Allowing more files than openssl is still "incompatible" in the sense that a golang app will behave differently than a native app.

Our solution is deployed with multiple parts in different languages and our customers can control certificates (so they can use mitm proxy) with the env variables.
So a customer that uses the SSL_CERT_DIR incorrectly and points it at a directory with a root ca bundle - it will work for the golang apps in the solution, but then fail for other apps.
If golang was strictly compatible, the "main" golang app wouldn't work and give a direct indicative error instead of failing somewhere down the line.

There is also some potential "security" consideration - as golang is trusting root CAs that the "system" doesn't trust and the user didn't mean to trust. Issue #39540 touches upon this, but with the "default" search that looks into /etc/pki/tls/certs/ - which also shouldn't just be blindly trusted. The same is true for SSL_CERT_DIR.

[seankhliao]

I don't think this is something we can or should change as it would be wildly disruptive to existing users.

We've never stated that the setting is bug-for-bug compatible with openssl. For Go, it's "override the system default locations for the SSL certificate file and SSL certificate files directory", and afaik there's no naming requirements for the system cert directory.

Given that it's possible to use the setting in a way that's compatible with applications using openssl, I think that's good enough. Note that other ecosystems typically also have their own cert lookup mechanisms, there's not really a single set of system certs on linux.