Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: Go 1.15 will break default connections to AWS RDS #39568

Closed
kevinburkemeter opened this issue Jun 12, 2020 · 30 comments
Closed

crypto/x509: Go 1.15 will break default connections to AWS RDS #39568

kevinburkemeter opened this issue Jun 12, 2020 · 30 comments
Assignees
Milestone

Comments

@kevinburkemeter
Copy link

@kevinburkemeter kevinburkemeter commented Jun 12, 2020

Amazon Relational Database Storage (RDS) is a popular database storage engine. RDS offers a method to securely connect to your database instance, described here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html.

RDS databases - Postgres ones, anyway, have not tested the other SQL flavors - send back a certificate that includes a common name field. Our own databases are inside of a VPN so it's a little trickier to share access for you to test them that way, but let me know if you need to and we can work something out.

/usr/local/opt/openssl@1.1/bin/openssl s_client -starttls postgres secretdatabasename.us-west-2.rds.amazonaws.com:5432
CONNECTED(00000005)
depth=2 C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
verify return:1
depth=1 C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
verify return:1
depth=0 CN = secretdatabasename.us-west-2.rds.amazonaws.com, OU = RDS, O = Amazon.com, L = Seattle, ST = Washington, C = US
verify return:1
---
Certificate chain
 0 s:CN = secretdatabasename.us-west-2.rds.amazonaws.com, OU = RDS, O = Amazon.com, L = Seattle, ST = Washington, C = US
   i:C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
 1 s:C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
   i:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
 2 s:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
   i:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA

There are no problems connecting using Go up through 1.14. However, if you try to connect to a RDS database using TLS and the Go 1.15 beta, this is the message that you get:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

I suspect that a lot of people are going to get caught out by this. (Well, maybe not, maybe people don't encrypt connections to the database.)

To be clear I think it is good that we are trying to get code (and certificates) to do the right thing. But it's not great that a bunch of code that previously worked will now not work.

I don't have the juice with AWS to ask them to update the certs to use SAN's. Perhaps someone reading this thread does have the juice, and can explain the problem and ask them to upgrade their certificates?

Could we call this out more clearly in the docs? For example, we could move the description of the change closer to the top of go1.15.html, or indicate that we expect that this change is going to break a lot of existing deployments, including RDS and possibly others.

@kevinburkemeter kevinburkemeter changed the title crypto/x509: Go 1.15 will break connections to AWS RDS crypto/x509: Go 1.15 will break default connections to AWS RDS Jun 12, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 12, 2020

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 12, 2020

Marking as a 1.15 release blocker to make sure that we make some sort of fix or decision.

@kevinburkemeter
Copy link
Author

@kevinburkemeter kevinburkemeter commented Jun 12, 2020

Amazon's response may be something along the lines of "this doesn't really matter if you are using our VPC since packets in a VPC are encrypted and authenticated" but you'd still need it if you were connecting to the database from outside of AWS.

@graywolf-at-work
Copy link

@graywolf-at-work graywolf-at-work commented Jun 12, 2020

Doesn't this kinda break https://golang.org/doc/go1compat? Or this this (not using common name) classify as a matter of security?

@mholt
Copy link

@mholt mholt commented Jun 13, 2020

@graywolf-at-work No, that's only a language guarantee. Old Go programs still compile and run. This is like phasing out support for SSLv3 and Windows XP, and is necessary to keep the standard library current with external changes.

@kaelanfouwels
Copy link

@kaelanfouwels kaelanfouwels commented Jun 13, 2020

CN verification is still specified by RFC 6125 as a fallback if SAN isn’t present.

Breaking external compatibility to prove a point seems excessive.

https://tools.ietf.org/html/rfc6125#section-6.4.4

@ulikunitz
Copy link
Contributor

@ulikunitz ulikunitz commented Jun 13, 2020

CL 231379 made "ignoreCN" the default. Reverting the first line of the diff, should bring back the old behavior.

@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Jun 13, 2020

It should be noted that this will only break non-publicly trusted, non-web PKI certificates since the CN is mandated in the CABF BRs to be, if present at all, one of the identities in the SAN, making this change non-breaking for conforming CAs.

CN was deprecated in 2000, and RFC 6125 made the fallback to it optional, rather a requirement (which was initially suggested in the RFC that deprecated it, RFC 2818).

@alex
Copy link
Contributor

@alex alex commented Jun 13, 2020

It seems like the most efficient path forward here is if someone from AWS can indicate their plans around adding SANs to RDS certificates.

@kevinburkemeter
Copy link
Author

@kevinburkemeter kevinburkemeter commented Jun 13, 2020

There's some indication on Twitter that the AWS team is looking at it but I'm not sure it will be ready by the release date. I don't know people in the right places at AWS to say otherwise. Filippo asked them for a timeline when they have one. https://twitter.com/seakoz/status/1271681101149270016

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Jun 13, 2020

The ideal resolution here is for AWS to use the correct field in their certificates. Let's wait to hear from them, and then we can decide whether to back off this change to Go 1.16.

Note that this change doesn't make it impossible to (manually) use CN for verification, it simply aligns with the spec in not considering it a hostname or an IP for VerifyHostname or VerifyOptions.DNSName purposes.

CN has been deprecated for 20 years, the specs do not recommend supporting it anymore, the browsers successfully dropped it, the WebPKI CAs are prohibited from setting it to anything that is not also in the SANs, and its behavior is completely unspecified.

The last point is a security issue with Name Constraint validation (which we support), because CN is not constrained (because it's not a hostname), so if we consider it a hostname we allow NCs to be bypassed. We have workarounds but they are complex unspecified hacks (see #24151).

@kevinburkemeter
Copy link
Author

@kevinburkemeter kevinburkemeter commented Jun 14, 2020

I'm just gonna say - I caught this because I deployed the 1.15 beta to our staging environment shortly after it was released. This gave us and AWS months of lead time between detection and the deadline for the release.

Please try to run the betas/RC's in your staging environments so that the entire community can benefit from the issues you find there, before they go live to production. Thanks!

@kse-clearhaus
Copy link

@kse-clearhaus kse-clearhaus commented Jun 17, 2020

We run a 3-D Secure Server. The certificates used by the scheme HTTPS endpoints
are universally issued by internal CAs.

This change will also break our connections to several major card schemes,
since they do not include SANs in certificates.

I'll grant you, that we will be able to work around it.
My point is, there are likely many areas where this causes issues.
Since there's actually an RFC that allows for a fallback, I would appriciate
if this was allowed.

Though I applaud your attempt to enforce a 20 year old deprecation :)

On a sidenote:

These CA's universally also encode nearly everything as ASN.1
PrintableString. This causes us heaps of trouble, much more than this would.

There's been several issues regarding this:
https://github.com/golang/go/issues?q=is%3Aissue+is%3Aclosed+printablestring

@FiloSottile FiloSottile self-assigned this Jun 23, 2020
@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jun 30, 2020

@FiloSottile If it's decided that this is going to be rolled back then it would be best to do it before RC1 is released.

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Jun 30, 2020

@FiloSottile If it's decided that this is going to be rolled back then it would be best to do it before RC1 is released.

I don't think this needs to be rolled back based only on the AWS RDS breakage.

I do think (hopefully incorrectly) that we'll see more breakage as more people test this, but if we roll it back based on my hunch of what will come out of RC1, then we're not actually learning what breaks and we'll never get to land this, because the same argument will apply unchanged for Go 1.16rc1.

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 17, 2020

Change https://golang.org/cl/243221 mentions this issue: doc/go1.15: surface the crypto/x509 Common Name deprecation note

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jul 23, 2020

As an update, we intend on releasing the RC with this change. There will be documentation in the release notes on how to get the old behavior while people work through issues. We're keeping this issue labeled as a release-blocker for Go 1.15, pending feedback from more people testing.

@luxifer
Copy link

@luxifer luxifer commented Jul 24, 2020

For the record I'm experiencing the same issue as described but with Cisco ISE. Certificates generated by ISE contains CN but no SAN thus the certificate is not validated by the go tls.

gopherbot pushed a commit that referenced this issue Jul 24, 2020
Updates #39568
Updates #37419
Updates #24151

Change-Id: I44c940e09e26a039076396bbfecb2b1574197cf7
Reviewed-on: https://go-review.googlesource.com/c/go/+/243221
Reviewed-by: Kevin Burke <kev@inburke.com>
@danp
Copy link
Contributor

@danp danp commented Jul 29, 2020

AWS have added a note to https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html about this issue:

If you are using a Go version 1.15 application with a DB instance that was created or updated to the rds-ca-2019 certificate prior to July 28, 2020, you must update the certificate again. Run the modify-db-instance command shown in the AWS CLI section using rds-ca-2019 as the CA certificate identifier. In this case, it isn't possible to update the certificate using the AWS Management Console. If you created your DB instance or updated its certificate after July 28, 2020, no action is required. For more information, see Go GitHub issue #39568

@jdorothy
Copy link

@jdorothy jdorothy commented Aug 21, 2020

Relating to what @cmars mentioned - as far as I can tell the AWS cert fix doesn't apply to, for example, GovCloud, which is still on rds-ca-2017. I don't know if that affects the decision to drop the workaround completely in Go 1.16, but it does put a lot of government projects in a bind.

@alex
Copy link
Contributor

@alex alex commented Aug 21, 2020

@jdorothy Have you confirmed that GovCloud still doesn't issue certs with SANs for RDS instances? It seems possible to me they use a different CA, but updated the issuing code to include SANs for it.

(I'm reasonably certain I recognize your name, which means we have some former colleagues we can probably ping to help nudge AWS along here.)

@jdorothy
Copy link

@jdorothy jdorothy commented Aug 21, 2020

@alex certainly didn't expect to see your name chiming in! Based on the output of openssl s_client -starttls postgres database:5432 < /dev/null | openssl x509 -noout -text it turns out GovCloud is indeed including SANs on new DBs:

        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:41:BB:F9:BA:FC:2A:4B:F0:3D:DE:60:EF:54:6F:87:6B:2D:72:52:5B
                DirName:/C=US/L=Seattle/ST=Washington/O=Amazon Web Services, Inc./OU=Amazon RDS/CN=Amazon RDS GovCloud Root CA
                serial:10:00

            X509v3 Subject Key Identifier:
                09:9A:13:D6:08:99:E5:DD:3E:7F:CA:A6:B5:F3:71:AD:79:E3:1F:16
            X509v3 Subject Alternative Name:
                DNS:dbname.us-gov-west-1.rds.amazonaws.com

Then I ran the same command against an older DB and sure enough, no SAN.

Solution
The solution is to run the exact same command linked in #39568 (comment), but replacing rds-ca-2019 with rds-ca-2017, even if you're already using rds-ca-2017. There appears to be a similar date cutoff as described in the AWS link, where new DBs are fine but DBs older than a certain date need to be updated (the DB I had to update was created 07-22-2020).

@alex
Copy link
Contributor

@alex alex commented Aug 21, 2020

👋 :-) Wonderful!

wallyqs added a commit to nats-io/nats-server that referenced this issue Sep 9, 2020
Starting at Go 1.15 now the TLS connections from Go clients
will fail with an error such as:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

when the certificates use the Common Name field.  As a workaround
to enable running tests with Go 1.15 (and also to demonstrate the
workaround for possibly impacted nats.go users in a commit), we
disable the client verification for the TLS map permissions.

More info on this at: golang/go#39568

Signed-off-by: Waldemar Quevedo <wally@synadia.com>
@wallyqs wallyqs mentioned this issue Sep 9, 2020
0 of 2 tasks complete
wallyqs added a commit to nats-io/nats-server that referenced this issue Sep 9, 2020
Starting at Go 1.15 now the TLS connections from Go clients
will fail with an error such as:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

when the certificates use the Common Name field.  As a workaround
to enable running tests with Go 1.15 (and also to demonstrate the
workaround for possibly impacted nats.go users in a commit), we
disable the client verification for the TLS map permissions.

More info on this at: golang/go#39568

Signed-off-by: Waldemar Quevedo <wally@synadia.com>
wallyqs added a commit to nats-io/nats-server that referenced this issue Sep 9, 2020
Starting at Go 1.15 now the TLS connections from Go clients
will fail with an error such as:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

when the certificates use the Common Name field.  As a workaround
to enable running tests with Go 1.15 (and also to demonstrate the
workaround for possibly impacted nats.go users in a commit), we
disable the client verification for the TLS map permissions.

More info on this at: golang/go#39568

Signed-off-by: Waldemar Quevedo <wally@synadia.com>
wallyqs added a commit to nats-io/nats-server that referenced this issue Sep 9, 2020
Starting at Go 1.15 now the TLS connections from Go clients
will fail with an error such as:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

when the certificates use the Common Name field.  As a workaround
to enable running tests with Go 1.15 (and also to demonstrate the
workaround for possibly impacted nats.go users in a commit), we
disable the client verification for the TLS map permissions.

More info on this at: golang/go#39568

Signed-off-by: Waldemar Quevedo <wally@synadia.com>
wallyqs added a commit to nats-io/nats-server that referenced this issue Sep 9, 2020
Starting at Go 1.15 now the TLS connections from Go clients
will fail with an error such as:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

when the certificates use the Common Name field.  As a workaround
to enable running tests with Go 1.15 (and also to demonstrate the
workaround for possibly impacted nats.go users in a commit), we
disable the client verification for the TLS map permissions.

More info on this at: golang/go#39568

Signed-off-by: Waldemar Quevedo <wally@synadia.com>
@wallyqs wallyqs mentioned this issue Sep 9, 2020
5 of 5 tasks complete
tidwall added a commit to tidwall/uhaha that referenced this issue Sep 29, 2020
Updated the tls configuration to support better SAN names and
phase out CN. golang/go#39568
StrongMonkey added a commit to StrongMonkey/linkerd2 that referenced this issue Oct 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.