-
Notifications
You must be signed in to change notification settings - Fork 781
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ECH (Encrypted client hello) support #1924
Comments
And the HPKE prerequisite on OpenSSL side was just merged in openssl/openssl@ad06248 🙂 |
Hiya, I did the PoC referred to above. While it'll be a while before that'd be ready, there are two things on which some discussion could be useful in the meantime:
I'd be happy to chat about either of the above as I work on a PR for OpenSSL for ECH. (That won't be done in a few weeks, but maybe early in the new year is reasonable.) |
Thank you Tristan, that's good news. We had discussions about this in the past but nothing was merged in master. HAProxy isn't well suited to handle "split-mode", because it will only be used as TCP-mode, and this mode is basically a tunnel once everything is connected, and handling HTTP connections with the TCP mode is not really recommended. We could discuss this more later but I think we should focus on the "shared" mode for now, which will be more easily integrated with the haproxy architecture. In my opinion the "shared" mode, aka HAProxy as a TLS endpoint, could be integrated in haproxy, I don't know what are the state of the patches at the moment, but tit looks like the fork is not up to date. |
Seconding William in that if one REALLY wants TCP mode haproxy + ECH they can use HAProxy as a pure « dumb » TCP forwarder (and yes it likely can’t do TCP SNI routing reliably in that case) and have whatever handles the actual TLS layer support the ECH bits. (And at that point you likely ought to be using something like LVS instead but that’s a whole other recurring topic because we all like haproxy too much to accept using something else :-) ) At least in my case only the so-called shared mode is really of interest, so it’s good that it’s the easier one 😄 |
wrt to @sftcd:
It's been a few billion years since I last wrote some C but I'll try and see if I can find time to trial your OpenSSL API proposal patches and get somewhere (in ~2-3 weeks likely) |
Chrome 117 support ECH now, any news on HAProxy ECH ? |
Certainly didn't have time to look into it myself just yet, unfortunately :/ In other news, I updated the tracking bug for FF as they now migrated they meta-issue from ESNI to ECH here: https://bugzilla.mozilla.org/show_bug.cgi?id=1725938 |
On 13/09/2023 07:35, 陈杨文 wrote:
Chrome 117 support ECH now, any news on HAProxy ECH ?
No news wrt releases from here, but we've kept our
OpenSSL ECH work up to date and interoperating with
browsers, including our integration with haproxy.
Details at [1].
We're thinking of making some PPAs available in the
future, so if that's of interest feel free to get in
touch.
Cheers,
S.
[1] https://defo.ie/
|
Fyi some more updates and links (since the Chrome shipping was spotted before I did :).
Since wolfSSL has support for it (when built with HPKE support), and given the woes with OpenSSL already, is it perhaps something to consider without waiting for OpenSSL? I ask because I am quite interested in it (and very willing to provide a testing ground for it, just like with QUIC before) and from the last few discussions on the ML and other issues here, there's less of a will to have HAProxy tied to OpenSSL than before and they seem intent on waiting until it is an RFC at least to consider it, which sftcd was guessing here would be at best Q1 2024. Also to note that when OpenSSL comes to support it, it will necessarily end up being a 3.1.x thing at best anyway... Which is going to be problematic wrt performance and distro availability (since the current distro generation is on 3.0.x mostly, and unlikely to ship 3.1.x before late-2024 at best...) |
We are running into issues utilizing HAProxy and our CloudFlare DNS as it appears CF is enforcing ECH by default causing issues with updated Chrome users. Is there a solution or work around to this? I am guessing HAProxy would need to support ECH or we would need to use something other than HAProxy till it is implemented? |
Not sure I understand the problem, but am interested - can you elaborate? |
We use CloudFlare for DNS and proxy on our production domains. I am unsure all the ins and outs of ECH but it seems there might be a DNS component that although our development subdomains are not proxied by Cloudflare ECH is still being enforced somehow. This is causing our older 2.1 instances of HAProxy to throw the error "ERR_ECH_FALLBACK_CERTIFICATE_INVALID" when a Chrome 217 client is trying to access one of those domains. I am guessing this is due to lack of ECH support in HAProxy? |
If Cloudflare forcefully publishes eager ECH records (HTTPS-type) on the zone forcefully, that sounds like it might cause issues indeed when clients trying ECH directly, even when they reach HAProxy without passing via CF. Easiest would be to have CF proxying for your environment too, or to see if overriding HTTPS record on your development domain’s zone fixes it. But since CF forcefully appends their CAA records and similar, I have a feeling this wouldn’t quite work… Maybe contact CF support asking them to opt-out your development domain zone from ECH. It’s likely they can do that. |
I think that we are having luck with the proxy, but not sure if that will change or not. We are forced to not proxy WSS connections as CF tends to break those from time to time causing connection issues for our users. So we are seeing those subdomains affected in our production environment... |
Thing is, even if HAProxy did support ECH, you would have the same issue. ECH essentially uses 2 different certs:
And the outer cert domain is defined by the DNS zone, and must be a valid cert (cloudflare-ech.com for Cloudflare iirc). The inner cert is the site-specific one you currently have. So that one is no problem. But naturally you wouldn’t be able to get such an outer cert yourself since you don’t control cloudflare-ech.com. I’m not familiar enough with the spec to know how HTTPS records deal with multiple possible outer certs and priorities though, but my guess is you’d need to set it up with a higher-priority record for a domain you control and generate an outer cert for. Edit: as clarified below by @sftcd, calling the identification material of an ECH exchange a "certificate" is incorrect |
We can see cloudflare-ech.com being queried in the failed requests so maybe there is some mechanism there to allow our downstream certs as a fallback? |
I wonder. You’ll have to bring it up with Cloudflare at this point. It’s not something HAProxy can help with directly (whether it supports ECH or not), as it’s a side effect of CF’s implementation of common zone records for their customers really. |
Don't have much time to comment now, but just to note that ECH does not involve use of new "certs" so the descriptions above are off-base a bit. |
My bad, let's say "key-pairs-of-some-sort" then. But unless I totally misunderstood, a relevant point is that you can't respond to an outer ClientHello requesting cloudflare-ech.com unless you have some private key material specific to that domain name, which only Cloudflare has as they publish the public portion of it on that zone, as far as I understand it. So the effect is analoguous unless CF were to make that material public, which I'm not sure they're even allowed to (just like publishing a cert private key voluntarily is forbidden by the CAB and will result in any such cert being revoked). |
Sorta. With ECH a browser will emit an initial TLS message (the ClientHello) that can only be fully processed by the entity with the relevant ECH private key. As I imagine CF deploy stuff, they assume that message will be sent to them. If somehow that message is sent somewhere else, then yes, ECH decryption won't happen, and the TLS handshake will fail with a (non-informative;-) message like the one quoted above. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This seems like a large oversight and not much education was put out to compensate for this massive change. There are TONs of people that just use CloudFlare for DNS.... I have opened a ticket and replied on 1 of 2 semi-related ECH forum posts in the CF community forums so here's hoping they have some "good" solution to give us. |
If you only use CF for DNS, then ECH shouldn't be in the picture. CF would have to be adding "ech=" values into your HTTPS RRs for that to happen. Are they doing that? If so, that'd be visble via |
For example:
Is for a CF customer (I think) but has no ech= and hence ECH will not be attempted. |
Most likely they do it on @ if it is set to proxy mode, and then it applies to subdomains? Because on a zone of mine I do indeed not see it yet. Tho rte.ie might be on a non-free CF plan and they're delaying the rollout for those. idk. But if for free CF accounts people are in a situation of:
have the second one falling back to using the HTTPS record on @? Then again, that sounds odd since HTTPS is more akin to an A/AAAA record than anything else, but idk... maybe just browser bug atm |
Not sure - I'm not familiar with CF's biz model(s), sorry. Doesn't (so far) sound like a browser bug though, but one could tell for a given DNS name using dig and wireshark pretty quickly, so shouldn't be hard to figure what behaviour is happening vs. expected. |
I do see the records everywhere here, might be your dig is outdated or something |
Not sure, just installed it via my WSL AlmaLinux 8 environment 🤷 |
I see the same
|
That said we strictly stick to DNS-only usage of CF and indeed these records don't show up yet on our zone so idk.
But since they do still generate TLS certs for our domain, and inject their own CAA records as well, I suspect that it is an oversight of theirs and we'll soon be in the same situation as you. So at this point we're just going to move out of CF for auth NS, just in case. Nonetheless, all of this should very much be a discussion with CF. |
Yep ageed, I appreciate you assisting me a bit in diagnosing this. Hopefully not downplaying this issue :) |
The Cloudflare issue is indeed off topic. I guess people will be on the lookout for a workaround and then may come back here posting: It looks like disabling TLSv1.3 in your Cloudflare settings works around this problem even on the free plan, so you avoid the need to upgrade to a paid plan just to restore DNS-only services. Source: https://community.cloudflare.com/t/getting-ech-fallback-certificate-error/565167/16 All further discussions regarding Cloudflare belong into their community forums (or support) of course. Let's keep this feature request on topic. |
Your Feature Request
ECH (Encrypted client hello) is a developing specification for encrypting the original client hello in an HTTP1.1/HTTP2 context.
The main purpose is making SNI sniffing impossible by middle boxes and other such adversarial systems. The practical implementation is not dissimilar to how SSL certificates trust is established, by using certain new DNS records (+ DNS-over-HTTPS/TLS) as source of public keys involved (instead of a few CAs).
I had a short chat on the topic with @wlallemand at HAProxyConf, and he was aware of it and of the PoC referenced below. He hinted at it maybe being less relevant than before due to QUIC bringing encryption all the way through, but QUIC reaching the same level of usage as HTTP1.1/2 will take years. Especially when it still relies on Alt-Svc response headers at the moment, and while one will soon be able to advertise QUIC at the DNS level directly (see https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https/) this is also going to take a hot minute to be widely available, so I'm still quite interested in ECH in general (and hopefully I'm not alone in that).
Some relevant references/notes:
https://bugzilla.mozilla.org/show_bug.cgi?id=1590863https://bugzilla.mozilla.org/show_bug.cgi?id=1725938More specifically for HAProxy, the work done by the DEfO PoC people has progressed quite a bit on the OpenSSL side:
This is still somewhat early days (need HPKE merged, ECH to go from draft to RFC, and OpenSSL to adopt ECH) but I thought I'd raise this issue to have it in the tracker.
What are you trying to do?
Use ECH with HAProxy
Output of
haproxy -vv
The text was updated successfully, but these errors were encountered: