crypto/x509: path lookup order change affects CentOS from Go 1.7->8

[joegrasse]

What version of Go are you using (go version)?

go version go1.8rc3 linux/386

What operating system and processor architecture are you using (go env)?

GOARCH="386"
GOBIN=""
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr2/ps/isp/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_386"
GCCGO="gccgo"
GO386=""
CC="gcc"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build005892966=/tmp/go-build"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

go get -d -u <package location>

What did you expect to see?

Successful completion of go get

What did you see instead?

package <package location>: unrecognized import path "<package location>" (https fetch: Get https://<package_url>?go-get=1: x509: certificate signed by unknown authority)

Using go version go1.7.4 linux/386, this worked successfully. We are using our own CA, and the package_url is our own internal server.

[joegrasse]

Doing more research looks like

go/src/crypto/x509/root_linux.go

Line 10 in ce64553

"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7

was added in go1.8. On our server we have both /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and /etc/pki/tls/certs/ca-bundle.crt. In go 1.7.4, it would find /etc/pki/tls/certs/ca-bundle.crt, which has the root certificate. In go 1.8rc3, it found /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem first, which doesn't have the root certificate.

Our server is CentOS 6.8.

[bradfitz]

If this were found during the Go 1.8 beta testing period, or even an earlier rc, it would've been easier to fix. It might be too late now. :-/

As a workaround, can you just add your certs to the first location?

Do other popular TLS clients read & merge all files?

/cc @agl

[joegrasse]

@bradfitz By default the openssl command was working correctly. It was able to verify the certificate.

[bradfitz]

But because it's looking at one location or merging multiple?

[joegrasse]

Here is a snippet from man verify.

"The process of ’looking up the issuers certificate’ itself involves a number of steps. In versions of OpenSSL before 0.9.5a the first certificate whose subject name matched the issuer of the current certificate was assumed to be the issuers certificate. In OpenSSL 0.9.6 and later all certificates whose subject name matches the issuer name of the current certificate are subject to further tests."

Which to me sounds like it keeps looking.

[agl]

I think that section of the man page is suggesting that it'll check multiple certificates from a file, as opposed to merging several files. Generally OpenSSL is configured with https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_load_verify_locations.html which takes a single file and a single directory for hash lookups. Of course, OpenSSL is very complex and it's possible to make things happen differently and there's the possibility of distro-specific patches too.

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

[joegrasse]

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

I not saying it doesn't, but with the change in go1.8, go isn't looking there by default.

I am still trying to research this, but it looks like in CentOS 6 (not sure what dot revision it was introduced) there is a legacy way and a new way to handle certs.

[joegrasse]

@bradfitz I am leaning toward this being a breaking change for CentOS 6, because I believe by default CentOS 6 uses the legacy method, which is /etc/pki/tls/certs/ca-bundle.crt. I am by no means an expert in this though.

[agl]

To clarify: does CentOS include both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem but expect that the former is the default?

[bradfitz]

If we got this bug report two months a fix would've been much easier.

Between go1.8beta1, go1.8beta2, go1.8rc1, go1.8rc2, and go1.8rc3 we make fewer and smaller and safer changes. We're basically not making changes at this point of the release cycle. We can't risk causing regressions for other crypto/x509 users. The fact that we only got one bug report about this (and so late) suggests this isn't an issue that will impact many people at least.

Unless there's a great diagnosis of what's happening and a super minimal and obviously-correct fix, you'll probably have to just modify both *.crt files, use a locally-patched Go, stay on Go 1.7, or wait for Go 1.9. Sorry.