Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kontena certificate authorize --type tls-sni-01 fails with challenge: LE gave back empty tls-sni-01 challenge!?!? #3209

Closed
SpComb opened this issue Jan 10, 2018 · 5 comments
Labels
Milestone

Comments

@SpComb
Copy link
Contributor

SpComb commented Jan 10, 2018

This is because LE has disabled all TLS-SNI based challenges as of January 10, 2018 02:16 UTC: https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/5a55777ed9a9c1024c00b241

This is a mitigation for a security weakness in how service providers do not sufficiently validate custom user-provided certificates deployed to shared infrastructure, allowing other users of the same provider to illegitimately validate domains owned by other customers of the same service provider: https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996

This affects automated LE certificate renewals by the Kontena server, and those certificates may expire unless LE either restores TLS-SNI support, or the cert domains are re-validated using some other method.

The current workaround for Kontena users is to use kontena certificate authorize --type dns-01, with either manual or automated mechanisms to update their DNS TXT records for renewals.

Long-term, it's looking fairly clear that Kontena needs to also support the more popular and widely used http-01 mechanism, which is far less likely to be unexpectedly disabled than the relatively unpopular tls-sni-* mechanisms.

$ kontena certificate authorize --type tls-sni-01 --linked-service ingress-lb/lb terom-demo3.kontena.works
 [error] 422 : Unprocessable Entity:
	challenge: LE gave back empty tls-sni-01 challenge!?!?
@SpComb
Copy link
Contributor Author

SpComb commented Jan 11, 2018

Current situation is that LE is now white-listing specific providers known to perform sufficient validation for certs. I don't think we're likely to see TLS-SNI challenges being available for use by Kontena users again at any time in the near future, and most likely not before LE certificates deployed by Kontena users will start failing to renew and expire.

We are currently working on fast-tracking http-01 support as a replacement for tls-sni-01: #3212

Update #1: We have decided to re-enable the TLS-SNI-01 challenge for certain major providers who are known not to have issues while we investigate re-enabling TLS-SNI-01 in general. We’re doing this as a safe way to restore service faster for a large number of sites.

@SpComb
Copy link
Contributor Author

SpComb commented Feb 27, 2018

Status Quo is that all tls-sni challenges have been permanently disabled for new accounts and new domain authorizations, but as a "temporary" mitigation for certificate renewals, tls-sni-01 domain authorizations are whitelisted for the specific accounts and domains that were previously authorized using tls-sni-01 before it was disabled:

This means that certificates requested using kontena certificate authorize --type tls-sni-01 should continue to auto-renew 7 days before expiry.

Testing for tls-sni-01 cert renewals

The ability to auto-renew certificates using existing tls-sni-01 challenges can be tested by attempting to re-authorize the same domain manually.

Fail: Using a new LE account

$ kontena certificate authorize --type tls-sni-01 --linked-service ingress-lb/lb 18.195.116.41.xip.io
 [error] 422 : Unprocessable Entity:
	challenge: LE did not offer any tls-sni-01 challenge

Success: Using the same LE account used to authorize the domain previously

$ kontena certificate authorize --type tls-sni-01 --linked-service ingress-lb/lb 18.195.116.41.xip.io
 [done] Waiting for tls-sni-01 challenge to be deployed into development/ingress-lb/lb      
TLS-SNI challenge certificate is deployed, you can now request the actual certificate
$ kontena certificate request 18.195.116.41.xip.io
 [done] Requesting certificate for 18.195.116.41.xip.io      
$ kontena certificate show 18.195.116.41.xip.io
development/18.195.116.41.xip.io:
  subject: 18.195.116.41.xip.io
  valid until: 2018-05-28T11:58:33Z
  auto renewable: true
$ kontena certificate export --cert 18.195.116.41.xip.io | openssl x509 -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fa:1a:dc:6f:c7:3c:1a:a7:63:cc:08:6c:e9:80:04:5e:4c:bf
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Fake LE Intermediate X1
        Validity
            Not Before: Feb 27 11:58:33 2018 GMT
            Not After : May 28 11:58:33 2018 GMT
        Subject: CN=18.195.116.41.xip.io

@SpComb SpComb added this to the 1.5.0 milestone Feb 27, 2018
@SpComb
Copy link
Contributor Author

SpComb commented Feb 27, 2018

Milestoning to 1.5 because this can be closed once 1.5.0 is available and kontena/docs#71 is merged, allowing users to move from tls-sni-01 to http-01.

@SpComb
Copy link
Contributor Author

SpComb commented Apr 6, 2018

Closing as 1.5 is available and defaults to http-01 challenges, and kontena/docs#71 should be up on the docs site soon.

@SpComb SpComb closed this as completed Apr 6, 2018
@SpComb
Copy link
Contributor Author

SpComb commented Apr 6, 2018

We should probably still keep tls-sni-01 support around for now, in order to support cert renewals... but at some point it can be deprecated and eventually removed, as it's no longer supported on ACME for new users. Existing users can migrate to http-01, which is a drop-in replacement in Kontena.

BTW, here's the post-disclosure writeup, it seems like domains hosted on Heroku and AWS CloudFront were affected: https://labs.detectify.com/2018/01/12/how-i-exploited-acme-tls-sni-01-issuing-lets-encrypt-ssl-certs-for-any-domain-using-shared-hosting/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant