-
Notifications
You must be signed in to change notification settings - Fork 7.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Istio does not adhere to HTTP/2 RFC 7540 #13589
Comments
Seems related: #9429 |
Seems like the spec says the we cannot return a 421? Or maybe I am reading that wrong |
@howardjohn they mean proxies in the traditional sense, istio could be considered a reverse proxy but its not a proxy by the RFC definition. Folks at chromium also clarified that the proxy only applies to forward proxies: https://bugs.chromium.org/p/chromium/issues/detail?id=954160#c5 (see second to last comment) |
@howardjohn thoughts? |
I agree with what you said, Istio isn't a proxy by the rfc definition Seems like a bug but probably something to change in Envoy rather than Istio? |
I can open an issue on envoy, link the tickets, we'll see where we land, for now if its alright to leave this ticket up to let other teams comment that would be helpful. |
@howardjohn If you'd like to expand what envoy would need to fix it would help, I've opened a ticket for them here: envoyproxy/envoy#6767 |
@trevorlinton could you share |
@PiotrSikora I'm unsure how to grab that, I'm not terribly familiar with envoy, just as much as exposed by istio. Here's the
|
@trevorlinton it's Could you also paste output from |
Sadly I removed our test cluster after I dumped the first configuration, I can give you instructions in Istio as to how to re-create it. |
@trevorlinton that would be great, thanks. |
Then visit a.example.com on Chrome or Firefox. Notice how the reference for b.example.com/foo.png returns a 404. Where the expected result is it should return a 421, the browser should re-negotiate the SNI and a new connection then the image should eventually be resolved and the site should successfully work. Notice that via curl you can retrieve both https://a.example.com/index.html and https://b.example.com/foo.png but in the browser going to https://b.example.com/foo.png fails (even when directly navigating to it) if you've first been to a.example.com and have it open in a different tab. Hope this helps replicate it. |
This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions. |
This issue has been automatically closed because it has not had activity in the last month and a half. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions. |
This issue should be reopened with an external ref envoyproxy/envoy#6767
— https://httpwg.org/specs/rfc7540.html#reuse @istio/wg-networking-maintainers |
A CVE was opened (not by me) surrounding this issue: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-11767 |
/reopen |
I still think this is relevant, and should not be closed, thank you. |
How other proxies handle this: Apache: https://httpd.apache.org/docs/2.4/mod/mod_http2.html#misdirected automatic detection of a vhost requested that has different TLS config https://tools.ietf.org/html/rfc6066#section-11.1 seems to imply we should be asserting SNI matches Host |
Hi, Do we have a WA for this issue? I've facing the exact issue with wildcard cert and 3 different services |
There are three choices to work around the issue. These focus on making sure the browser doesn't try to reuse your connections:
The browser reuses connections if the IP:Port is the same as a previous connection and the previous connection's cert is valid for the new hostname. See the HTTP spec for details. If you can affect one of those 3 items (IP, Port, Cert), you can avoid the connection reuse, which works around Istio's limitation. |
We ran into this issue and ended up disabling HTTP/2 in istio with the following config. apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: disable-alpn-h2
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: FILTER_CHAIN
match:
listener:
filterChain:
sni: "*.mygateway.com"
patch:
operation: MERGE
value:
transportSocket:
name: envoy.transport_sockets.tls
typedConfig:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
commonTlsContext:
alpnProtocols:
- "http/1.1"
tlsCertificateSdsSecretConfigs:
- name: kubernetes://wildcard-cert
sdsConfig:
ads: {}
resourceApiVersion: V3 |
@ragingpastry this not work for me. cause
@howardjohn can we expose the AlpnProtocols config to
now we have to reorder the AlpnProtocols fields after the listener patches, this is so tricky way. Look forward to your reply. thanks! @howardjohn |
Perhaps someone could take this issue to a Networking WG meeting? |
This has been discussed a few times in the past and we were unable to achieve consensus on a path forward. I think @lambdai has a doc about this we can probably attach for more context |
how can I tell the chrome disable http2 such as "chrome --disable-http2" , or tell the istio gateway protocol: HTTPS1.1 |
Looking forward to having this resolved. |
Is there someone working on this? 😳 |
Thanks for the excellent writeup. I think we're seeing the same issue with a slightly different setup. (Please tell me if I'm wrong. I'm happy to open a separate case.)
Unrelated differences (I think?)
We also noticed that we can not reproduce this behavior in the latest Firefox. Maybe it's being more conservative and choosing not to re-use http/2 connections across hostnames? (because of just these kinds of server-side errors? Or because it's simpler not to? 🤷) This gave us a ton of headaches because we started experiencing this when we were adding and testing CORS headers to allow requests across those domains. Whether that would work really depended on the particular access pattern that might trigger the bad behavior. And we couldn't see anything wrong in our apps because the gateway was just rejecting the routes before even sending them on to our pods. I estimate we wasted a cumulative week of developer time (across multiple developers) if not more because of this issue. Is there nothing to be done that can at least avoid this issue until there's a fix in envoy? (For example: Maybe the gateway could refuse to reuse HTTP/2 connections on certs that are wildcards and/or serve multiple hostnames? at least until the upstream fix is available?) Our workaround for now was to create separate certs for each hostname. This is fine in development, but isn't really tenable (as people say above) if you have a large number of hostnames to deal with, and can't act as your own CA. |
This bug is biting me right now, looks like I can relatively easily switch to non-overlapping certs. A bummer wildcard certs dont work with http/2 it seems? |
The easiest quick and dirty solution would be to allow users to disable ALPN for HTTP/2 on the gateway, because right now it's hard-coded to be on with no settings to force disable it. |
Any updates on this would be appreciated. We rely on wildcard certificates quite a lot and this currently makes it very hard to get istio running stable enough for production. Having the option to disable HTTP/2 (even if that's not the default behaviour) would be really welcome. |
I would appreciate any updates on this topic too as we are hit in our development environment by this issue. This problem appeared after we migrated away from traefik. |
I had the same problem, when using two services, one which is part of istio-servicemesh and one which is not. Both with the same wildcard-certificate. For me it seems to be working fine when creating an additional catch-all virtual service with wildcard-host on the related gateway.
I found the needed information here: |
Bug description
Istio does not properly send a 421 response when a connection is reused and accident sent to a server that is not the correct origin. This can occur when there are two gateways, one with a wildcard certificate (*.example.com) and one with a different non-wildcard certificate (b.example.com) routing to two different apps (a.example.com and b.example.com) where a http/2 connection is first established to the wildcard gateway (on host a.example.com with *.example.com) then a resource is requested from an application on the non-wildcard gateway (b.example.com with certificate b.example.com).
Because of http/2 connection reuse it's possible for traffic destined for the second app (b.example.com) to end up being routed on the existing connection for (a.example.com) due to the RFC definition of connection re-use in section 9.1.1 (https://tools.ietf.org/html/rfc7540#section-9.1.1, e.g., because the certificate can authoritatively handle the request, and the IP address is the same as they are on the same ingressgateway).
When that happens, according to section 9.1.2 istio should respond with a 421 indicating that the wrong connection was used and the origin was not found. This would instruct the browser to retry on a new connection, thus renegotiating TLS and presenting SNI and thus going down the non-wildcard certificate route and to a different gateway/virtual service and the correct service.
Expected behavior
Should return a 421 and the browser should re-connect and successfully find the resource. Instead a 404 is returned.
Steps to reproduce the bug
https://b.example.com/foobar.png
(e.g.,<img src="https://b.example.com/foobar.png">
)Version (include the output of
istioctl version --remote
andkubectl version
)How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
AWS, with istio installed and running with an NLB ingress or as a nodeport terminating the TLS.
The text was updated successfully, but these errors were encountered: