Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

Closed
joegrasse opened this issue Jan 27, 2017 · 27 comments
Closed

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

joegrasse opened this issue Jan 27, 2017 · 27 comments
Assignees
Milestone

Comments

@joegrasse
Copy link

@joegrasse joegrasse commented Jan 27, 2017

What version of Go are you using (go version)?

go version go1.8rc3 linux/386

What operating system and processor architecture are you using (go env)?

GOARCH="386"
GOBIN=""
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr2/ps/isp/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_386"
GCCGO="gccgo"
GO386=""
CC="gcc"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build005892966=/tmp/go-build"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

go get -d -u <package location>

What did you expect to see?

Successful completion of go get

What did you see instead?

package <package location>: unrecognized import path "<package location>" (https fetch: Get https://<package_url>?go-get=1: x509: certificate signed by unknown authority)

Using go version go1.7.4 linux/386, this worked successfully. We are using our own CA, and the package_url is our own internal server.

@joegrasse joegrasse changed the title cmd/go: Regression with go get of package cmd/go: Regression with go get of package from go1.7.4 to go1.8rc3 Jan 27, 2017
@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

Doing more research looks like

"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7
was added in go1.8. On our server we have both /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and /etc/pki/tls/certs/ca-bundle.crt. In go 1.7.4, it would find /etc/pki/tls/certs/ca-bundle.crt, which has the root certificate. In go 1.8rc3, it found /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem first, which doesn't have the root certificate.

Our server is CentOS 6.8.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 27, 2017

If this were found during the Go 1.8 beta testing period, or even an earlier rc, it would've been easier to fix. It might be too late now. :-/

As a workaround, can you just add your certs to the first location?

Do other popular TLS clients read & merge all files?

/cc @agl

@bradfitz bradfitz changed the title cmd/go: Regression with go get of package from go1.7.4 to go1.8rc3 crypto/x509: go get regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change Jan 27, 2017
@bradfitz bradfitz added this to the Go1.8Maybe milestone Jan 27, 2017
@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

@bradfitz By default the openssl command was working correctly. It was able to verify the certificate.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 27, 2017

But because it's looking at one location or merging multiple?

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

Here is a snippet from man verify.

"The process of ’looking up the issuers certificate’ itself involves a number of steps. In versions of OpenSSL before 0.9.5a the first certificate whose subject name matched the issuer of the current certificate was assumed to be the issuers certificate. In OpenSSL 0.9.6 and later all certificates whose subject name matches the issuer name of the current certificate are subject to further tests."

Which to me sounds like it keeps looking.

@agl
Copy link
Contributor

@agl agl commented Jan 27, 2017

I think that section of the man page is suggesting that it'll check multiple certificates from a file, as opposed to merging several files. Generally OpenSSL is configured with https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_load_verify_locations.html which takes a single file and a single directory for hash lookups. Of course, OpenSSL is very complex and it's possible to make things happen differently and there's the possibility of distro-specific patches too.

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

I not saying it doesn't, but with the change in go1.8, go isn't looking there by default.

I am still trying to research this, but it looks like in CentOS 6 (not sure what dot revision it was introduced) there is a legacy way and a new way to handle certs.

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

@bradfitz I am leaning toward this being a breaking change for CentOS 6, because I believe by default CentOS 6 uses the legacy method, which is /etc/pki/tls/certs/ca-bundle.crt. I am by no means an expert in this though.

@agl
Copy link
Contributor

@agl agl commented Jan 27, 2017

To clarify: does CentOS include both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem but expect that the former is the default?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 27, 2017

If we got this bug report two months a fix would've been much easier.

Between go1.8beta1, go1.8beta2, go1.8rc1, go1.8rc2, and go1.8rc3 we make fewer and smaller and safer changes. We're basically not making changes at this point of the release cycle. We can't risk causing regressions for other crypto/x509 users. The fact that we only got one bug report about this (and so late) suggests this isn't an issue that will impact many people at least.

Unless there's a great diagnosis of what's happening and a super minimal and obviously-correct fix, you'll probably have to just modify both *.crt files, use a locally-patched Go, stay on Go 1.7, or wait for Go 1.9. Sorry.

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

@bradfitz I have already worked around the issue, I am more just worried about the CentOS 6 user that this might break. What population of them have tested go1.8?

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

To clarify: does CentOS include both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem but expect that the former is the default?

@agl I believe Yes and Yes. I don't have a "stock" CentOS 6 box though.

@agl
Copy link
Contributor

@agl agl commented Jan 27, 2017

It sounds like we could just flip the order of the two entries then, but I'd love to get some clarification from RedHat about their policy here. I'll see if I can reach someone who'll claim to know.

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 27, 2017

@bradfitz @agl I am wondering for easier discovery and because this doesn't just impact go get if the title should now be

crypto/x509: "certificate signed by unknown authority" regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change.

@bradfitz bradfitz changed the title crypto/x509: go get regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change crypto/x509: path lookup order change affects CentOS from Go 1.7->8 Jan 27, 2017
@agl
Copy link
Contributor

@agl agl commented Jan 27, 2017

(I've emailed @nmav to see if we can get word about what the "correct" answer is here.)

@nmav
Copy link

@nmav nmav commented Jan 28, 2017

In RHEL7 the file to use for openssl apps is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem [0]. ca-bundle.crt is a link to the first one.

In RHEL6 the primary file is ca-bundle.crt. (There, it is optional for an administrator to enable the rhel7 trust management, by running update-ca-trust enable. In that case tls-ca-bundle.pem and ca-bundle.crt should be the same file).

Based on the above I think using ca-bundle.crt seems like a safe choice.

[0]. http://www.unix.com/man-page/centos/8/update-ca-trust/

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 30, 2017

@agl As a side note, I have noticed the following.

CentOS 7

~$ ls -alh /etc/pki/tls/cert.pem
lrwxrwxrwx. 1 root root 49 Nov 29 02:53 /etc/pki/tls/cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

CentOS 6

~$ ls -alh /etc/pki/tls/cert.pem
lrwxrwxrwx. 1 root root 19 Jun  8  2016 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt

Not sure if it would make sense to just point at /etc/pki/tls/cert.pem?

@joegrasse
Copy link
Author

@joegrasse joegrasse commented Jan 31, 2017

@bradfitz @agl It looks like ce64553 is the commit that caused this problem. It also looks like it was a fix for #15749. On CentOS 7 though, /etc/pki/tls/certs/ca-bundle.crt is linked to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem. So, I think there was a different cause for the issue in #15749.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Feb 1, 2017

I defer to @agl to decide what to do here.

@agl, we ship Feb 14th. It'd be nice to have this done before then. There might be an -rc4.

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 2, 2017

CL https://golang.org/cl/36093 mentions this issue.

@rsc
Copy link
Contributor

@rsc rsc commented Feb 7, 2017

I agree that CL 36093 is too much for Go 1.8. Can we just move the new RHEL7 line down to the bottom of the preference list for Go 1.8?

ping @agl

@rsc rsc modified the milestones: Go1.8, Go1.8Maybe Feb 7, 2017
@rsc
Copy link
Contributor

@rsc rsc commented Feb 7, 2017

Sent CL 36429 to reorder the list. I checked, and the line was added for #17549, which was "can't find any certs"; that is, on the system we were trying to fix, none of the listed files existed. On this CentOS 6 system, the existing files were better than the new file, but putting the new file earlier in the list broke things. Moving the new file to the end of the list should keep #17549 fixed and still fix this issue.

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 7, 2017

CL https://golang.org/cl/36429 mentions this issue.

@agl
Copy link
Contributor

@agl agl commented Feb 7, 2017

It's still unclear why /etc/pki/tls/certs/ca-bundle.crt didn't exist in #15749, but Russ's change should still take care of that so LGTM.

@gopherbot gopherbot closed this in 1ead0bd Feb 7, 2017
@rsc
Copy link
Contributor

@rsc rsc commented Feb 7, 2017

https://go-review.googlesource.com/36530 for cherry-pick to Go 1.8.

@rsc rsc reopened this Feb 7, 2017
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 7, 2017

CL https://golang.org/cl/36530 mentions this issue.

@agl agl assigned rsc and unassigned agl Feb 8, 2017
gopherbot pushed a commit that referenced this issue Feb 8, 2017
We added CentOS 7's /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
to the list in response to #17549 - not being able to find any certs otherwise.

Now we have #18813, where CentOS 6 apparently has both that file
and /etc/pki/tls/certs/ca-bundle.crt, and the latter is complete while
the former is not.

Moving the new CentOS 7 file to the bottom of the list should fix both
problems: the CentOS 7 system that didn't have any of the other files
in the list will still find the new one, and existing systems will still
keep using what they were using instead of preferring the new path
that may or may not be complete on some systems.

Fixes #18813.

Change-Id: I5275ab67424b95e7210e14938d3e986c8caee0ba
Reviewed-on: https://go-review.googlesource.com/36429
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Adam Langley <agl@golang.org>
Reviewed-on: https://go-review.googlesource.com/36530
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@rsc rsc modified the milestones: Go1.9, Go1.8 Feb 8, 2017
@rsc
Copy link
Contributor

@rsc rsc commented Feb 8, 2017

Closing. CL 36093 mentions this issue but is really about a broader thing. This specific issue should now be fixed entirely.

@rsc rsc closed this Feb 8, 2017
gopherbot pushed a commit that referenced this issue May 3, 2017
Add the ability to override the default file and directory from
which certificates are loaded by setting the OpenSSL compatible
environment variables: SSL_CERT_FILE, SSL_CERT_DIR.

If the variables are set the default locations are not checked.

Added new default file "/usr/local/etc/ssl/cert.pem" for FreeBSD.

Certificates in the first valid location found for both file and
directory are added, instead of only the first file location if
a valid one was found, which is consistent with OpenSSL.

Fixes #3905
Fixes #14022
Fixes #14311
Fixes #16920
Fixes #18813 - If user sets SSL_CERT_FILE.

Change-Id: Ia24fb7c1c2ffff4338b4cf214bd040326ce27bb0
Reviewed-on: https://go-review.googlesource.com/36093
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@golang golang locked and limited conversation to collaborators Feb 8, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.