New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Envoy endpoints not loading since Chrome release 124 #33850
Comments
cc @ggreenway who may know more about the different TLS protocols that are supported. |
I would find it shocking if chrome wouldn't be willing to negotiate one of the older TLS 1.3 ciphers in this case; I'd guess that most TLS endpoints on the internet don't yet support the post-quantum cipher suites. Regardless, I think if there's an issue here, it's a bug in chrome. |
The problem has made some news website like MSN, I'm not sure what's the policy about posting links so I'll post an excerpt
|
I might have found some helpful comment on DDG
I think I saw a setting that affects this size in Envoy |
I can't find such option, any suggestions? |
Can you capture a full tcpdump of the failed handshake and post it? |
@ggreenway it doesn't seem to accept pcap files, any suggestions? |
Ignore that, here's the gzipped pcap |
Huh, it looks like the tcp window is closed after only 1400 bytes. Can you post the full envoy configuration you used for this test? |
1400 bytes seems like a typical MTU, maybe we're limiting handshake to a single packet? |
Here's a cut-down version of our config, I hope I didn't axe too much
|
https://tldr.fail/ describes the issue. It is likely that Envoy is not reading the entire Client Hello as it spans packets. |
@vparla thanks for that. I used the linked python script to test against the latest Envoy and it worked correctly (Envoy sent back a ServerHello). @cd-fernando can you try the script at https://github.com/dadrian/tldr.fail/blob/main/tldr_fail_test.py and see what it returns? Can anyone else test this and report success or failure? |
I got one more report of it working correctly with Chrome. I think I understand what's going on: the TlsInspector filter doesn't read from the socket; it peeks. This means the entire ClientHello needs to fit into the configured socket read buffer size. I saw in the tcpdump that the server had a fully filled up tcp window (of only about 1500 bytes), and I didn't realize until now that it was the TlsInspector, not the TLS transport socket, that was getting stuck. I've never seen a socket read buffer configured that small. What OS are you using? If you don't need to select a filter chain based on SNI, you can remove the TlsInspector from your config and that should fix this. |
Thanks everyone for looking into this. For a bit more background in case you're curious it's a value we had set to optimise Haproxy and that only, unfortunately because our QA boxes are self contained, that setting affected Envoy too. It had never been a problem until the enabling of these Quantum resistant protocols. Thank you very much again! FYI that script doesn't seem to work on versions of Python older than 3.11. |
Thank you very much for your assistance, I'm closing this now. |
Just so I can document this on tldr.fail---where is the socket read buffer configured in this context? Is that an Envoy, TLS Inspector, or kernel setting (or somewhere else)? I would have expected that to be internal to Envoy's implementation, but it didn't seem like this needed a code change? |
@dadrian it's the kernel socket receive buffer. On linux it's normally set with |
Title: Envoy not responding since release of Chrome 124
Description:
As part of the release of Chrome version 124, Google seems to have widely enabled post-quantum secure TLS key encapsulation Kyber768 for TLS 1.3
Repro steps:
In Chrome flags chrome://flags/ check if "TLS 1.3 hybridized Kyber support" is enabled
From Chrome, connect to any TLS endpoint served by Envoy, and it gets stuck on a spinning wheel until if fails.
I would appreciate any suggestions on this issue, I am not well versed on Envoy.
The text was updated successfully, but these errors were encountered: