Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLS-ALPN-01 support for Certbot #6724

Open
enoch85 opened this issue Jan 30, 2019 · 23 comments
Open

TLS-ALPN-01 support for Certbot #6724

enoch85 opened this issue Jan 30, 2019 · 23 comments

Comments

@enoch85
Copy link

@enoch85 enoch85 commented Jan 30, 2019

I think this post sums it up quite well. Simple question; when will TLS-ALPN-01 become available in Certbot?

Many people rely on Certbot and thousands of users got an email warning them to change method from TLS-SNI. Since Certbot is recommended by Let's Encrypt isn't it in your interest to make it available for everyone using Certbot as well?

Thanks!

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Jan 30, 2019

Closing as a duplicate of #6724. Thanks for the input!

@ohemorange ohemorange closed this Jan 30, 2019
@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Jan 30, 2019

Ah, I see this is a slightly different issue.

@ohemorange ohemorange reopened this Jan 30, 2019
@ohemorange ohemorange changed the title When will TLS-ALPN-01 become available in Certbot? TLS-ALPN-01 support for Certbot Jan 30, 2019
@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Jan 30, 2019

This issue will follow after #6676.

@enoch85

This comment has been minimized.

Copy link
Author

@enoch85 enoch85 commented Feb 4, 2019

So will this be in the 0.31.0 milestone?

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Feb 4, 2019

Unlikely; we will try to fit it in our roadmap though.

@methodbox

This comment has been minimized.

Copy link

@methodbox methodbox commented Feb 5, 2019

Can you clarify? My understanding was Certbot could not support this in the automated fashion used with TLS-SNI-01 due to something with Apache/Nginx servers which does not allow the communication protocol?

@ohemorange When this is implemented, will we be able to automatically renew using a similar method to what we currently use with Certbot & TLS-SNI-01 on Apache & Nginx servers?

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Feb 6, 2019

To be clear, this issue is only intending to implement support for the standalone plugin, not the Apache or Nginx plugins.

It is currently possible to use the HTTP-01 challenge type with the Apache and Nginx plugins in place of the TLS-SNI-01 challenge type to automatically renew certs. Perhaps this is about using port 80, in which case I recommend this documentation page on the Let's Encrypt website.

@methodbox

This comment has been minimized.

Copy link

@methodbox methodbox commented Feb 7, 2019

To be clear, this issue is only intending to implement support for the standalone plugin, not the Apache or Nginx plugins.

It is currently possible to use the HTTP-01 challenge type with the Apache and Nginx plugins in place of the TLS-SNI-01 challenge type to automatically renew certs. Perhaps this is about using port 80, in which case I recommend this documentation page on the Let's Encrypt website.

Yeah, it was actually but thanks for the reply anyway.

The problem with your "solution" to just leave port 80 open is that in the corporate environment network engineers like to put various sites only on non-web ports meaning we would have to add a load balancer or something similar to proxy these types of requests.

While I agree that may indeed be the best way to handle it while also having other benefits, the powers that be do not for various reasons.

Unfortunately, that means using a paid certificate instead of Let's Encrypt. I understand corporate networks may not be your target audience, but just my two cents.

Thanks for clarifying.

@adferrand

This comment has been minimized.

Copy link
Contributor

@adferrand adferrand commented Feb 7, 2019

Hello @methodbox ! Corporate environments are just more difficult to handle, because of their inherent complexity. However, efforts are done on our side to better match theses particular needs.

As you seem to have a corporate need, I would like to take the occasion to better understand you case, as I use also Certbot in corporate environments.

It is true that supporting TLS-ALPN-01 on Apache and Nginx is very difficult, because these HTTP servers do not support this TLS extension natively. Purely theoretically, I was thinking of alternatives ways to still support a TLS-ALPN-01 challenge around a running Nginx/Apache server. One of them would imply to place temporarily a proxy in front of the HTTP server to serve specifically the challenge. However, this would imply at the beginning and at the end of the challenge (not during it), a short outage of few ms, maybe 1s, do make internal switch of the listening ports.

Would it be something acceptable in the absolute ?

A little off-topic also, but to let you check the possibility: have you considered DNS-01 challenges ? They fit well in a corporate environment, and allow wildcard certificates that are really helpful for these cases.

@methodbox

This comment has been minimized.

Copy link

@methodbox methodbox commented Feb 8, 2019

Hello @methodbox ! Corporate environments are just more difficult to handle, because of their inherent complexity. However, efforts are done on our side to better match theses particular needs.

As you seem to have a corporate need, I would like to take the occasion to better understand you case, as I use also Certbot in corporate environments.

It is true that supporting TLS-ALPN-01 on Apache and Nginx is very difficult, because these HTTP servers do not support this TLS extension natively. Purely theoretically, I was thinking of alternatives ways to still support a TLS-ALPN-01 challenge around a running Nginx/Apache server. One of them would imply to place temporarily a proxy in front of the HTTP server to serve specifically the challenge. However, this would imply at the beginning and at the end of the challenge (not during it), a short outage of few ms, maybe 1s, do make internal switch of the listening ports.

Would it be something acceptable in the absolute ?

A little off-topic also, but to let you check the possibility: have you considered DNS-01 challenges ? They fit well in a corporate environment, and allow wildcard certificates that are really helpful for these cases.

Technically speaking, the best solution, as it applies to web applications that are on non-standard ports (i.e. not 80/443), in my opinion, is to use NGINX as a load balanacing proxy. This has a bunch of other benefits and doesn't stop the internal sites from being accessed via the non-standard port directly.

As an example, you could accept the request via the load balancer, but still make the site available via domain.com:10002 directly. This is beneficial in that the current operating procedures do not need to change (people don't need to change old habbits for the new configuration) but that it effectively solves the issue.

Hypothetically, it's also possible you only actually enable this proxy server at times when you need to renew certificates, if that's it's only purpose and use scheduled configurations in your firewall to allow it's connection (exa. the proxy server is started between 8pm - 10pm on the 1st of each month and the port schedule in the firewall is set to do the same) which limits the attack vector to virtually nothing for those paranoid about port 80.

The problem with this is usually that the people in charge either don't understand and/or simply do not want to deal with the complexity of executing such a configuration. Which is ironic, considering the complexity is an effect of their current design decisions with the existing network as it is.

So no, for me, it wouldn't be acceptable but that's not due to a technical limitation at all.

Regarding the DNS challenge, while I suggested we manage DNS programmatically via the API offered by the registrar (which has potential to allow us to automate the process) this is also not an option. As it stands, by design, the DNS option is not able to be automated without further configuration.

For what it's worth @ohemorange you should understand the appeal to Let's Encrypt has less to do with the cost and a LOT to do with automation. It's just my view, but you certainly have a marketing opportunity with your users to enable this kind of thing for paid certificates.

When you have to install 300 certificates and cannot allow a single cert to expire prematurely the appeal of automation cannot be overstated.

@adferrand

This comment has been minimized.

Copy link
Contributor

@adferrand adferrand commented Feb 8, 2019

For the hypothesis I had, @methodbox, it was a dynamic deployment of a proxy in an automated way. Currently, apache and nginx plugins are already instrumenting automatically the existing configuration of the running HTTP server to make it able to serve the challenge, then provide the created certificate on HTTPS connections. So the goal is still, for theses plugins, to make things automatically, not requiring the user to make manual operations, except the execution of Certbot itself.

@methodbox

This comment has been minimized.

Copy link

@methodbox methodbox commented Feb 8, 2019

For the hypothesis I had, @methodbox, it was a dynamic deployment of a proxy in an automated way. Currently, apache and nginx plugins are already instrumenting automatically the existing configuration of the running HTTP server to make it able to serve the challenge, then provide the created certificate on HTTPS connections. So the goal is still, for theses plugins, to make things automatically, not requiring the user to make manual operations, except the execution of Certbot itself.

I'm sorry; not sure if you're asking a question or explaining, but I understand your statement.

The problem in our case is multiple web servers serving applications on non-standard (not 80/443) ports. Each of these MUST listen on a non-standard port because they are on unique, private network IPs to which traffic is routed via a single public IP by a firewall. This means all traffic comes in on a single IP from the firewall, but is routed effectively based on which port is passed at the end of the URL. It's sort of a poor man's load balancer via firewall (it's not even using the load balancing capabilities of the firewall).

This means a load balancer or proxy capable of reading the HTTP header would be necessary (a la web server proxy) to separate that traffic to the proper machine and proxy the requests from the Let's Encrypt ACME server from port 80 to the respective private IP:port.

It also means opening port 80 for the public IP on the firewall.

Under this configuration, the proxy server could be made to respond and house the certificates and that solves our issue.

Alternatively, a port 80 listener could be added to each of the respective servers to respond to their respective domain names, however, this means all port 80 requests would go to all three servers and they would all have to field unnecessary traffic and load.

While I suggested this as an option, it seems the primary concern by the user is even opening port 80 in the first place.

@bmw bmw added this to the 0.33.0 milestone Feb 20, 2019
@bmw

This comment has been minimized.

Copy link
Member

@bmw bmw commented Mar 4, 2019

When we resolve this issue, we might want to revisit #6742.

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Mar 5, 2019

When doing this, we should check that openssl won't put "acme-tls/1" in ServerHello "without actually understanding [the protocol]". https://tools.ietf.org/html/draft-ietf-acme-tls-alpn-01

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Mar 5, 2019

To summarize an OOB conversation: since the ACME server validates that the ACME client serves a certificate over a TLS handshake that includes “acme-tls/1” in ServerHello, we should actually be fine even given the known OpenSSL implementation gotchas.

@bmw

This comment has been minimized.

Copy link
Member

@bmw bmw commented Mar 19, 2019

The previous code for adding support for this to our ACME code can be found at #6100.

In addition to any merge conflicts, another issue that needs to be fixed in that code is:

...in some versions of openssl, we're hitting issues where the challenge cert is used even if the CA requests the wrong protocol through ALPN.

This behavior itself isn't a problem, but some of our unit tests aren't expecting this behavior and failing. I don't remember what tests were failing, but I've described the change in behavior and how to reproduce it at pyca/pyopenssl#769.

Adding support for TLS-ALPN-01 to Certbot's standalone plugin should probably be done in a subsequent PR after support in acme lands, but one thing to keep in mind when adding this support is we cannot unconditionally offer TLS-ALPN-01 support in standalone. We should do checks similar to those found in the PR linked above (perhaps by putting the checks in a function standalone can call) to only include TLS-ALPN-01 in the preferred challenges list returned by the plugin if OpenSSL has the support we need.

@dholth

This comment has been minimized.

Copy link

@dholth dholth commented Mar 25, 2019

I'm working on an acme responder in twisted. It's easy to parse clienthello in any openssl version because twisted uses bio. That means all input/output goes through Python before being piped into OpenSSL, so we see the plain bytes of ClientHello as the first packet. I'd appreciate feedback or help.

glyph/txsni#28

@bmw bmw modified the milestones: 0.33.0, 0.34.0 Apr 3, 2019
@bmw bmw modified the milestones: 0.34.0, 0.35.0 Apr 15, 2019
@alexander255

This comment has been minimized.

Copy link

@alexander255 alexander255 commented Apr 28, 2019

Mostly out of curiosity: Would it be possible to support the TLS-ALPN-01 with nginx using the $ssl_preread_alpn_protocols variable of the ngx_stream_ssl_preread_module (in particular see the second example on that site)? It looks like that could be used to deflect acme-tls/1 ALPN TLS connections to a standalone server running on a different port maybe?

Not saying that this is important in the short run, just curious if that would be “enough support” from nginx's side to potentially enable this use-case.

@bmw bmw added priority: normal and removed current sprint labels Apr 29, 2019
@methodbox

This comment has been minimized.

Copy link

@methodbox methodbox commented May 18, 2019

Mostly out of curiosity: Would it be possible to support the TLS-ALPN-01 with nginx using the $ssl_preread_alpn_protocols variable of the ngx_stream_ssl_preread_module (in particular see the second example on that site)? It looks like that could be used to deflect acme-tls/1 ALPN TLS connections to a standalone server running on a different port maybe?

Not saying that this is important in the short run, just curious if that would be “enough support” from nginx's side to potentially enable this use-case.

TL;DR: Yes, and no.

The stream pre-read for SSL is effectively for allowing SSL passthrough on a load balancer; the certificate doesn't need to be available on the load balancer, but since it can read the ClientHello it can determine which domain/upstream to proxy it to and allow that server to terminate TLS; i.e. the cert only has to exist on that server.

It's really what I was saying above; you can have your service on whatever port you want if you either using a load balancer or some configuration that can proxy that request to the proper port, but you need to be able to field the ALPN request.

But as @ohemorange has expressed this doesn't alleviate the need for a solution that would allow the standard automation previously enabled by the [now defunct due to security] SNI protocol method.

In theory, that NGINX plugin could allow usage of the ALPN protocol in such a way as to respond specifically to an ACME server request, but it isn't implemented that way directly.

You may want to have a look @lukas2511 's package "dehydrated" - https://github.com/lukas2511/dehydrated/blob/master/docs/tls-alpn.md

He has an example implementing this, but it's a "hacked together solution" (by his own words) for doing this using his module. His example doesn't use the upstream load balancing function, which I believe is what you were targeting by your description, but I'm sure it could be adapted.

We chose not to got this route as it's untested, unsupported and the package hasn't been updated in >10 months. We don't have time to do a security assessment for implementing something like this vs. using a standard cert until a solution is provided by Let's Encrypt.

@dholth

This comment has been minimized.

Copy link

@dholth dholth commented May 18, 2019

I had fun doing txsni+alpn, here's an asyncio version: https://bitbucket.org/dholth/aiosni/src/default/aiosni/__init__.py . It sends a different certificate when the client requests acme-tls/1 as an ALPN protocol. It could be used to build a web server that also works together with something like dehydrated to get certificates for itself, as opposed to a server that always sends the acme-tls/1 certificate and never sends the normal certificate.

@bmw bmw modified the milestones: 0.35.0, 0.36.0, 0.37.0 Jun 3, 2019
@bmw bmw modified the milestones: 0.37.0, 0.38.0 Jul 15, 2019
@bmw bmw removed this from the 0.38.0 milestone Aug 12, 2019
@enoch85

This comment has been minimized.

Copy link
Author

@enoch85 enoch85 commented Nov 1, 2019

Any progress on this?

@adferrand

This comment has been minimized.

Copy link
Contributor

@adferrand adferrand commented Dec 29, 2019

Any progress on this?

First step of the implementation (adding necessary classes in acme module) is under review here: #6886.

@koen-serneels

This comment has been minimized.

Copy link

@koen-serneels koen-serneels commented Dec 30, 2019

I've used another alternative for TLS-ALPN-01, might be interesting to check out if you're on Apache at least: mod_md. The latest version is now by rfc8555 (ACMEv2) complete and implements OCSP stapling. Although not yet included in the latest Apache 2.4 build (according to the notes it will in the next one) it's rather easy to compile from source. Configuration is also easy, integrates nicely with the rest of the sites configuration, no port 80 needed/no downtime for cert renewal, overview of certificate status via de status-page (if enabled) etc

bmw pushed a commit that referenced this issue Mar 12, 2020
This PR is the first part of work described in #6724.

It reintroduces the tls-alpn-01 challenge in `acme` module, that was introduced by #5894 and reverted by #6100. The reason it was removed in the past is because some tests showed that with `1.0.2` branch of OpenSSL, the self-signed certificate containing the authorization key is sent to the requester even if the ALPN protocol `acme-tls/1` was not declared as supported by the requester during the TLS handshake.

However recent discussions lead to the conclusion that this behavior was not a security issue, because first it is coherent with the behavior with servers that do not support ALPN at all, and second it cannot make a tls-alpn-01 challenge be validated in this kind of corner case.

On top of the original modifications given by #5894, I merged the code to be up-to-date with our `master`, and fixed tests to match recent evolution about not displaying the `keyAuthorization` in the deserialized JSON form of an ACME challenge.

I also move the logic to verify if ALPN is available on the current system, and so that the tls-alpn-01 challenge can be used, to a dedicated static function `is_available` in `acme.challenge.TLSALPN01`. This function is used in the related tests to skip them, and will be used in the future from Certbot plugins to trigger or not the logic related to tls-alpn-01, depending on the OpenSSL version available to Python.

* Reimplement TLS-ALPN-01 challenge and standalone TLS-ALPN server from #5894.

* Setup a class method to check if tls-alpn-01 is supported.

* Add potential missing parameter in validation for tls-alpn

* Improve comments

* Make a class private

* Handle old versions of openssl that do not terminate the handshake when they should do.

* Add changelog

* Explicitly close the TLS connection by the book.

* Remove unused exception

* Fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.