New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crypto/x509: "certificate is not standards compliant" on MacOS #51991
Comments
The error seems be introduced here: feb024f#diff-9e2a37df9605e8b207365b51999e6b14e1f5db72b27ad33514dbac502d477c25R212 @liggitt summarized the ask here : kubernetes/kubernetes#108956 (comment) and kubernetes/kubernetes#108956 (comment) on what impact this will have on kubernetes users for whom there was no issues before we switched to go1.17 when they try the kubernetes kubectl command built using go1.18. Worst case we would like to document scenarios under which users will hit the thanks! |
cc @golang/security |
Can you share the actual affected certificate? You are unlikely to be hitting the same root cause as |
We only noticed the issue in a unit test checking a cert we expected to be considered invalid (it was a negative test of a cert not signed by a trusted root), so at first we thought we just needed to make the unit test more tolerant of various error messages. In spelunking around the change in the message, we also ran across https://groups.google.com/g/golang-nuts/c/RGghq2gTWss/m/7GsudTfCAgAJ which indicated requests that previously succeeded were now failing. Before papering over the change in our unit test by tolerating more validation error messages, I wanted to understand more about which certificates the go validator considers valid that the macOS validator does not. (kubernetes/kubernetes#108956 (comment)) |
Apple enforces their SCT requirements on all publicly trusted certificates as part of its base TLS policy (which we use via This will only affect users who are using the bare system certificate pool and are validating certificates which totally lack embedded SCT, with the server providing them via a TLS extension. It is noted in a TODO in |
Slight side note: the AWS case is a weird one, because I don't think they are sending SCTs at all, despite using publicly trusted certs, so even implementations that know how to pipe SCTs passed via TLS extensions wouldn't work on macOS |
I don't think the SCT policy explains the new error in kubernetes/kubernetes#108956 though, because that's not a publicly trusted certificate. If you want to extract that certificate and share it with us, we can tell you why that one is failing, too.
We can't really answer this exhaustively, because the macOS verifier has a number of evolving policies that change between OS versions. Note that the platform verifier is only used when the system roots are involved, so behaving like the system is what's expected. I assume k8s clusters use private CAs configured through |
Customers should probably reach out to AWS about this. As a short term workaround, it should be possible to add the Amazon root CAs to a |
It looks like this change means we no longer get typed TLS errors (e.g. That means that special handling of those errors (logging or other fallback paths) that previously worked no longer works in go1.18. edit: I'll open a separate issue for that, since that is distinct from the "certificate considered valid in go1.17" → "certificate considered invalid in go1.18" issue |
opened #52010 for the untyped error issue |
I also expect that to be true in most scenarios (and in scenarios where it isn't for the certs issued by public CAs to be compatible with system roots, though the referenced AWS issue is evidence that not all certs issued by public CAs are valid). For k8s' use, I don't think this issue is very significant. |
Oh, that particular test certificate (in |
We've had the same issue with connecting to AWS Elasticache Redis servers. Amazon will not support SCTs to avoid publishing customer cluster names in a public log. The connection previously worked fine in 1.17. |
A little more background about AWS, or at least how we were connecting to the Neptune graph database service. In case it helps anyone. Because Neptune is a little "light" on security, you can only connect to it through local/private VPC. This isn't very useful for developers, so we have a VPN to a bastion host for a development only instance of Neptune (Neptune does not have any local installation - it is an AWS service only). It seems that AWS did not feel the need to put any SCTs in to the Neptune cert, thinking it would only see connections from the secured VPC, and so our connections (via go, it is fine from Java for instance) will fail. We have raised a ticket with AWS about this. There isn't much can be done about that in Go. As this is a developer only connection, we have created a reverse proxy with a local CA root. This allows the connection for developers. Hokey, but does what we want for a developer connection. The real solution is of course for AWS to re-issue their certificate, however they say they don't want SCTs in order to avoid placing customer cluster names in a public log (see #51991 (comment) ) |
This is showing up on the Marking as release-blocker for Go 1.19 because
2022-04-19T23:20:21-8b900b4-104742f/darwin-amd64-10_15 |
As far as I can tell this seems, possibly, (this is unbearably painful to diagnose) to be an issue with 10.15.1, which is what the the darwin-amd64-10_15 builder is running. I suspect that updating the builder to use 10.15.6 would fix this, but I have absolutely no clue how viable that is. |
@golang/release, can you weigh in on that? (How hard is the macOS 10.15 image to update?) |
For amd64 I think it's maybe a day's work, if we're willing to cut over all the builders at once. Rolling it out gradually will be more unpleasant. I haven't read this issue to judge whether it's a good use of time. |
I don't think there is really any other way to address this issue, given how deeply integrated the TLS client is in the toolchain there isn't really any (safe) way of silently handling/skipping these failures. It's not a high frequency flake though (it seems somewhat correlated with when new certificates are issued) so probably not super high priority. |
Can the machine reach the internet? That sounds consistent with a bloom filter window miss on the Apple Valid system, which leads to an OCSP connection to the CA. If that fails, I could see it leading to a vague error like this. |
In some cases certificates may be deliberately excluded from CT logs to avoid publishing a detailed map of internal infrastructure. (In our case, the certs are associated with DNS names that are only resolvable internally, and which resolve to private IPs.) e.g. AWS ACM allows disabling CT logging for this purpose, which will result in a valid certificate issued by a trusted CA but not listed in CT logs. When trying to access a service with such a cert from Go (in our case, using a Terraform provider) on developer ( Is there any solution other than logging the certs? Is there any knob in the Go TLS client for turning off the check, which the TF provider could provide a config option to turn? |
In our case, we were just needing developers to access a development
version of Neptune on AWS. We were doing that with a bastion host and ssh.
The Neptune cert is set up exactly as you say, to avoid public logs for
what is intended to be a system accessed from a VPC in AWS. We therefore
wrote a small reverse proxy instead of a simple tunnel. That works for our
case as our code runs in the VPC as AWS intend, other than for developers.
As I understand it from the comments here, Go 1.19 will try to address
this, but I cannot speak for the Go development team of course.
…On May 11, 2022 at 12:39:06 PM, Jasper ***@***.***> wrote:
In some cases certificates may be deliberately excluded from CT logs to
avoid publishing a detailed map of internal infrastructure. e.g. AWS ACM
allows disabling CT logging for this purpose, which will result in a valid
certificate issued by a trusted CA but not listed in CT logs.
When trying to access a service with such a cert from Go (in our case,
using a Terraform provider) on developer (darwin_arm64) machines, we get
this certificate is not standards compliant error.
Is there any solution other than logging the certs? Is there any knob in
the Go TLS client for turning off the check, which the TF provider could
provide a config option to turn?
—
Reply to this email directly, view it on GitHub
<#51991 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJ7TMBSJBB4IPNDNOYUWPLVJM2WVANCNFSM5R4DNYCQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hello: i can confirm that Amazon Neptune is affected :
what is the current recommend workaround? |
Amazon Elastic Cache Redis is affected as well. |
oc login --token=sha256~xxx --server=https://api.xx.com
error: x509: “api.xx.com” certificate is not standards compliant I just downgraded the |
Commenting to follow along - I started seeing CMS/PKS7 signatures failing on Darwin amd64 that use Digicert's RFC3161 Timestamp Server (http://timestamp.digicert.com)
I was able to manually work around by fetching the Digicert TS Crt from https://knowledge.digicert.com/content/dam/digicertknowledgebase/DigiCertTrustedRootG4.crt and adding it to an |
workaround: golang/go#51991
workaround: golang/go#51991
Pulumi had an internal user encounter this issue making requests to We're still keeping an eye on this. Has anyone else seen this behavior on Big Sur (11.x) and resolved by a patch release between 11.6 and 11.7? Alternatively, does anyone know if the updates to the macOS system are resolving the issue, or is the update clearing some cached state that would cause it to reoccur on the next certificate rotation? |
If this is helpful, I just saw this error trying to connect to a GKE cluster from MacOS 12.3.1, so I assume it was clearing the cache that did it. Incidentally, if there's a way to manually do this without updating the OS, it'd be useful to know as a workaround! |
For anyone struggling with this issue. I had the same problem:
Upgrading macOS Monterrey from 12.5 to 12.6.2 solved the issue for me!! It also seems to work with macOS 13 Ventura. I can now run the above program just fine with |
I am using go version go1.20.1 darwin/arm64 and MacOS Version 13.2.1 (22D68) and this issue is now occurring when I use github.com/gorilla/websocket .Dial() function. The error is : tls: failed to verify certificate: x509: “*.exchange.coinbase.com” certificate is not standards compliant" When is this scheduled to be resolved? Is there any simple workaround if accessing a public site where one has no control over the certificate? Any help greatly appreciated! |
@jameskilroe hey! we at Pulumi dug deep into this after seeing a similar issue. You should be able to resolve it by restarting your machine. We found that should almost always fix the issue. WorkaroundWe found that in every case where a user reported this issue, either of these were true:
In both cases, restarting the OS was the workaround. AnalysisI'm going to summarize my colleague @kmosher's analysis. In recent years Apple, Google, and others have added the following requirements for a TLS cert to be considered valid
On macOS, and in Safari and Go programs, the system service Due to unknown reasons, The trusted certificate transparency logs on macOS can be located here: /System/Library/Security/Certificates.bundle/Contents/Resources/TrustedCTLogs.plist And Apple publishes a JSON document for the OS to update from here: |
@rsc, @ianlancetaylor: Based on the above, I believe this issue can be closed - or perhaps moved to a Discussion with an accepted answer. I believe that someone should file a report with Apple's bug reporting tool Feedback Assistant to resolve this behavior with macOS if it hasn't been. This issue may already be fixed and released, but as we describe above, users who encounter the issue are likely to be on out of date machines that have not updated or even restarted their machine recently. |
Hi @AaronFriel Thanks for the answer. I updated my machine to the latest OS to try and solve the problem and did restart my machine a few time this morning, but unfortunately, the problem persists. I did notice that the files in my TrustedCTLogs.plist were last updated on 9 Feb (before my update). Is there any chance you know how to force and update? I did some googling but no obvious answers. All round very frustrating! |
@jameskilroe rather than looking at the date, diff it against the current list (https://valid.apple.com/ct/log_list/current_log_list.json). The set of valid CT logs doesn't change that often (last change 15 Jan). If you have the latest list and have rebooted recently, then you may have a different issue (like the certificate is not actually present in the CT logs — you can check this — or is valid for too long). |
Clients that rely on OSX APIs for certificate validation may find an error with the message "certificate is not standards compliant" with certificates that don't comply with Apple rules for certificate validation. When this error happens, the actual reason for each certificate is not exposed, and it seems to happen with certificates that should be valid in the context of the validation. More discussion about this can be found in golang/go#51991. This happens with certificates generated by `elastic-package`, clients sometimes report this error with configurations that otherwise should accept these certificates. According to this post, one of the rules is that certificates cannot be valid for more than 825 days. https://rahulkj.github.io/openssl,/certificates/2022/09/09/self-signed-certificates.html Reduce the expiration time to try to reduce the chances of triggering this error.
We hit an error with a unit test we had in Kubernetes and started looking at the impact on end users of kubernetes if the problem is not resolved by the time kubernetes 1.24 is released. More context: please see Kubernetes issue - kubernetes/kubernetes#108956
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Please see https://go.dev/play/p/w4rr43vQv7d
What did you expect to see?
success
What did you see instead?
error: x509: “no-sct.badssl.com” certificate is not standards compliant
The text was updated successfully, but these errors were encountered: