TLS-ALPN-01 support for Certbot

[enoch85]

I think this post sums it up quite well. Simple question; when will TLS-ALPN-01 become available in Certbot?

Many people rely on Certbot and thousands of users got an email warning them to change method from TLS-SNI. Since Certbot is recommended by Let's Encrypt isn't it in your interest to make it available for everyone using Certbot as well?

Thanks!

[ohemorange]

Closing as a duplicate of #6724. Thanks for the input!

[ohemorange]

Ah, I see this is a slightly different issue.

[ohemorange]

This issue will follow after #6676.

[enoch85]

So will this be in the 0.31.0 milestone?

[ohemorange]

Unlikely; we will try to fit it in our roadmap though.

[methodbox]

Can you clarify? My understanding was Certbot could not support this in the automated fashion used with TLS-SNI-01 due to something with Apache/Nginx servers which does not allow the communication protocol?

@ohemorange When this is implemented, will we be able to automatically renew using a similar method to what we currently use with Certbot & TLS-SNI-01 on Apache & Nginx servers?

[ohemorange]

To be clear, this issue is only intending to implement support for the standalone plugin, not the Apache or Nginx plugins.

It is currently possible to use the HTTP-01 challenge type with the Apache and Nginx plugins in place of the TLS-SNI-01 challenge type to automatically renew certs. Perhaps this is about using port 80, in which case I recommend this documentation page on the Let's Encrypt website.

[methodbox]

To be clear, this issue is only intending to implement support for the standalone plugin, not the Apache or Nginx plugins.

It is currently possible to use the HTTP-01 challenge type with the Apache and Nginx plugins in place of the TLS-SNI-01 challenge type to automatically renew certs. Perhaps this is about using port 80, in which case I recommend this documentation page on the Let's Encrypt website.

Yeah, it was actually but thanks for the reply anyway.

The problem with your "solution" to just leave port 80 open is that in the corporate environment network engineers like to put various sites only on non-web ports meaning we would have to add a load balancer or something similar to proxy these types of requests.

While I agree that may indeed be the best way to handle it while also having other benefits, the powers that be do not for various reasons.

Unfortunately, that means using a paid certificate instead of Let's Encrypt. I understand corporate networks may not be your target audience, but just my two cents.

Thanks for clarifying.

[adferrand]

Hello @methodbox ! Corporate environments are just more difficult to handle, because of their inherent complexity. However, efforts are done on our side to better match theses particular needs.

As you seem to have a corporate need, I would like to take the occasion to better understand you case, as I use also Certbot in corporate environments.

It is true that supporting TLS-ALPN-01 on Apache and Nginx is very difficult, because these HTTP servers do not support this TLS extension natively. Purely theoretically, I was thinking of alternatives ways to still support a TLS-ALPN-01 challenge around a running Nginx/Apache server. One of them would imply to place temporarily a proxy in front of the HTTP server to serve specifically the challenge. However, this would imply at the beginning and at the end of the challenge (not during it), a short outage of few ms, maybe 1s, do make internal switch of the listening ports.

Would it be something acceptable in the absolute ?

A little off-topic also, but to let you check the possibility: have you considered DNS-01 challenges ? They fit well in a corporate environment, and allow wildcard certificates that are really helpful for these cases.

[methodbox]

Hello @methodbox ! Corporate environments are just more difficult to handle, because of their inherent complexity. However, efforts are done on our side to better match theses particular needs.

As you seem to have a corporate need, I would like to take the occasion to better understand you case, as I use also Certbot in corporate environments.

It is true that supporting TLS-ALPN-01 on Apache and Nginx is very difficult, because these HTTP servers do not support this TLS extension natively. Purely theoretically, I was thinking of alternatives ways to still support a TLS-ALPN-01 challenge around a running Nginx/Apache server. One of them would imply to place temporarily a proxy in front of the HTTP server to serve specifically the challenge. However, this would imply at the beginning and at the end of the challenge (not during it), a short outage of few ms, maybe 1s, do make internal switch of the listening ports.

Would it be something acceptable in the absolute ?

A little off-topic also, but to let you check the possibility: have you considered DNS-01 challenges ? They fit well in a corporate environment, and allow wildcard certificates that are really helpful for these cases.

Technically speaking, the best solution, as it applies to web applications that are on non-standard ports (i.e. not 80/443), in my opinion, is to use NGINX as a load balanacing proxy. This has a bunch of other benefits and doesn't stop the internal sites from being accessed via the non-standard port directly.

As an example, you could accept the request via the load balancer, but still make the site available via domain.com:10002 directly. This is beneficial in that the current operating procedures do not need to change (people don't need to change old habbits for the new configuration) but that it effectively solves the issue.

Hypothetically, it's also possible you only actually enable this proxy server at times when you need to renew certificates, if that's it's only purpose and use scheduled configurations in your firewall to allow it's connection (exa. the proxy server is started between 8pm - 10pm on the 1st of each month and the port schedule in the firewall is set to do the same) which limits the attack vector to virtually nothing for those paranoid about port 80.

The problem with this is usually that the people in charge either don't understand and/or simply do not want to deal with the complexity of executing such a configuration. Which is ironic, considering the complexity is an effect of their current design decisions with the existing network as it is.

So no, for me, it wouldn't be acceptable but that's not due to a technical limitation at all.

Regarding the DNS challenge, while I suggested we manage DNS programmatically via the API offered by the registrar (which has potential to allow us to automate the process) this is also not an option. As it stands, by design, the DNS option is not able to be automated without further configuration.

For what it's worth @ohemorange you should understand the appeal to Let's Encrypt has less to do with the cost and a LOT to do with automation. It's just my view, but you certainly have a marketing opportunity with your users to enable this kind of thing for paid certificates.

When you have to install 300 certificates and cannot allow a single cert to expire prematurely the appeal of automation cannot be overstated.

[adferrand]

For the hypothesis I had, @methodbox, it was a dynamic deployment of a proxy in an automated way. Currently, apache and nginx plugins are already instrumenting automatically the existing configuration of the running HTTP server to make it able to serve the challenge, then provide the created certificate on HTTPS connections. So the goal is still, for theses plugins, to make things automatically, not requiring the user to make manual operations, except the execution of Certbot itself.

[methodbox]

For the hypothesis I had, @methodbox, it was a dynamic deployment of a proxy in an automated way. Currently, apache and nginx plugins are already instrumenting automatically the existing configuration of the running HTTP server to make it able to serve the challenge, then provide the created certificate on HTTPS connections. So the goal is still, for theses plugins, to make things automatically, not requiring the user to make manual operations, except the execution of Certbot itself.

I'm sorry; not sure if you're asking a question or explaining, but I understand your statement.

The problem in our case is multiple web servers serving applications on non-standard (not 80/443) ports. Each of these MUST listen on a non-standard port because they are on unique, private network IPs to which traffic is routed via a single public IP by a firewall. This means all traffic comes in on a single IP from the firewall, but is routed effectively based on which port is passed at the end of the URL. It's sort of a poor man's load balancer via firewall (it's not even using the load balancing capabilities of the firewall).

This means a load balancer or proxy capable of reading the HTTP header would be necessary (a la web server proxy) to separate that traffic to the proper machine and proxy the requests from the Let's Encrypt ACME server from port 80 to the respective private IP:port.

It also means opening port 80 for the public IP on the firewall.

Under this configuration, the proxy server could be made to respond and house the certificates and that solves our issue.

Alternatively, a port 80 listener could be added to each of the respective servers to respond to their respective domain names, however, this means all port 80 requests would go to all three servers and they would all have to field unnecessary traffic and load.

While I suggested this as an option, it seems the primary concern by the user is even opening port 80 in the first place.

[bmw]

When we resolve this issue, we might want to revisit #6742.

[ohemorange]

When doing this, we should check that openssl won't put "acme-tls/1" in ServerHello "without actually understanding [the protocol]". https://tools.ietf.org/html/draft-ietf-acme-tls-alpn-01

[ohemorange]

To summarize an OOB conversation: since the ACME server validates that the ACME client serves a certificate over a TLS handshake that includes “acme-tls/1” in ServerHello, we should actually be fine even given the known OpenSSL implementation gotchas.

[bmw]

The previous code for adding support for this to our ACME code can be found at #6100.

In addition to any merge conflicts, another issue that needs to be fixed in that code is:

...in some versions of openssl, we're hitting issues where the challenge cert is used even if the CA requests the wrong protocol through ALPN.

This behavior itself isn't a problem, but some of our unit tests aren't expecting this behavior and failing. I don't remember what tests were failing, but I've described the change in behavior and how to reproduce it at pyca/pyopenssl#769.

Adding support for TLS-ALPN-01 to Certbot's standalone plugin should probably be done in a subsequent PR after support in acme lands, but one thing to keep in mind when adding this support is we cannot unconditionally offer TLS-ALPN-01 support in standalone. We should do checks similar to those found in the PR linked above (perhaps by putting the checks in a function standalone can call) to only include TLS-ALPN-01 in the preferred challenges list returned by the plugin if OpenSSL has the support we need.

[dholth]

I'm working on an acme responder in twisted. It's easy to parse clienthello in any openssl version because twisted uses bio. That means all input/output goes through Python before being piped into OpenSSL, so we see the plain bytes of ClientHello as the first packet. I'd appreciate feedback or help.

glyph/txsni#28

[ntninja]

Mostly out of curiosity: Would it be possible to support the TLS-ALPN-01 with nginx using the $ssl_preread_alpn_protocols variable of the ngx_stream_ssl_preread_module (in particular see the second example on that site)? It looks like that could be used to deflect acme-tls/1 ALPN TLS connections to a standalone server running on a different port maybe?

Not saying that this is important in the short run, just curious if that would be “enough support” from nginx's side to potentially enable this use-case.

[methodbox]

Mostly out of curiosity: Would it be possible to support the TLS-ALPN-01 with nginx using the $ssl_preread_alpn_protocols variable of the ngx_stream_ssl_preread_module (in particular see the second example on that site)? It looks like that could be used to deflect acme-tls/1 ALPN TLS connections to a standalone server running on a different port maybe?

Not saying that this is important in the short run, just curious if that would be “enough support” from nginx's side to potentially enable this use-case.

TL;DR: Yes, and no.

The stream pre-read for SSL is effectively for allowing SSL passthrough on a load balancer; the certificate doesn't need to be available on the load balancer, but since it can read the ClientHello it can determine which domain/upstream to proxy it to and allow that server to terminate TLS; i.e. the cert only has to exist on that server.

It's really what I was saying above; you can have your service on whatever port you want if you either using a load balancer or some configuration that can proxy that request to the proper port, but you need to be able to field the ALPN request.

But as @ohemorange has expressed this doesn't alleviate the need for a solution that would allow the standard automation previously enabled by the [now defunct due to security] SNI protocol method.

In theory, that NGINX plugin could allow usage of the ALPN protocol in such a way as to respond specifically to an ACME server request, but it isn't implemented that way directly.

You may want to have a look @lukas2511 's package "dehydrated" - https://github.com/lukas2511/dehydrated/blob/master/docs/tls-alpn.md

He has an example implementing this, but it's a "hacked together solution" (by his own words) for doing this using his module. His example doesn't use the upstream load balancing function, which I believe is what you were targeting by your description, but I'm sure it could be adapted.

We chose not to got this route as it's untested, unsupported and the package hasn't been updated in >10 months. We don't have time to do a security assessment for implementing something like this vs. using a standard cert until a solution is provided by Let's Encrypt.

[dholth]

I had fun doing txsni+alpn, here's an asyncio version: https://bitbucket.org/dholth/aiosni/src/default/aiosni/__init__.py . It sends a different certificate when the client requests acme-tls/1 as an ALPN protocol. It could be used to build a web server that also works together with something like dehydrated to get certificates for itself, as opposed to a server that always sends the acme-tls/1 certificate and never sends the normal certificate.

[enoch85]

Any progress on this?

[adferrand]

Any progress on this?

First step of the implementation (adding necessary classes in acme module) is under review here: #6886.

[koen-serneels]

I've used another alternative for TLS-ALPN-01, might be interesting to check out if you're on Apache at least: mod_md. The latest version is now by rfc8555 (ACMEv2) complete and implements OCSP stapling. Although not yet included in the latest Apache 2.4 build (according to the notes it will in the next one) it's rather easy to compile from source. Configuration is also easy, integrates nicely with the rest of the sites configuration, no port 80 needed/no downtime for cert renewal, overview of certificate status via de status-page (if enabled) etc

[ndilieto]

You might also find this proxying tls-alpn-01 challenge responder interesting. It can respond to tls-alpn-01 challenges while at the same time transparently proxying normal HTTPS connections. It can work with any webserver and any ACME client.

Edit: I've made an experimental certbot plugin: https://github.com/ndilieto/certbot-ualpn

[gstrauss]

FYI: if you're looking for web servers which support TLS-ALPN-01:
Caddy supports TLS-ALPN-01 since Dec 2018 (https://caddy.community/t/caddy-supports-the-acme-tls-alpn-challenge/4860)
lighttpd 1.4.53 supports TLS-ALPN-01 since Jan 2019 (https://redmine.lighttpd.net/issues/2931)

I am a lighttpd developer and successfully use dehydrated (https://github.com/dehydrated-io/dehydrated) with TLS-ALPN-01.

I would love to see certbot add support for TLS-ALPN-01, too.