Repository navigation
0DPE (Downstream Protocol Error) caused by HTTP2 Filter #25218
Description
Bug description
I don't even know where to start with this one... I have no idea what went wrong but I'm going to share what info I do have here in case it gets anyones attention because it was quite a significant failure which in the end was recovered by restarting the source workload (nginx).
I know correlation != causation, but we've been stable on 1.5.0-1.5.6 for some time, without issue, and upgraded to 1.5.7 4 days ago.
Symptoms:
We started to see waves of request fail between two specific workloads on the cluster (all other workloads were fine). The source workload being nginx and the destination workload being consumer-gateway.
The errors came in waves of decreasing duration, you can see the scatter plot from Jaeger here how you get a spike, they gradually fail faster before eventually recovering.
There was no correlation on the source workload:
The errors primarily manifested in two ways:
1. 503UC from the source sidecar
This represented the majority of errors recorded, a 503 was recorded from the source (nginx) sidecar. There was no span recorded from the destination (consumer-gateway) leading me to believe the request was never even sent.
2. 0DPE
There was also a fair few of these "Downstream Protocol Errors" recorded by the destination (consumer-gateway) sidecar. These are interesting because it shows the request reached consumer-gateway
So far I've checked the following:
- There was nothing in pilot logs, nothing in the istio-proxy logs (although we run those in warn).
- There was no correlation from an infrastructure perspective (we observed failures across several nodes, across all zones). The only correlation was the fact the issue was between just these two workloads.
- We have rafts of network testing which indicated no issue.
- I wondered if this could be something odd to do with mtls cert rotation, but doesn't look like the certificates on either have changed recently:
❯ k -n consumer-gateway get secret istio.default -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -noout -text | grep Before
Not Before: Jun 18 17:57:06 2020 GMT
❯ k -n ingress-nginx get secret istio.default -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -noout -text | grep Before
Not Before: Jun 18 17:53:33 2020 GMT
Both workloads have been running for 4 days, since we upgraded to 1.5.7 (which is actually the only thing which has changed recently).
It's worth nothing we use both mTLS and http2 (via EnvoyFilter):
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: consumer-gateway
namespace: consumer-gateway
spec:
host: app
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 2147483646
http2MaxRequests: 2147483646
idleTimeout: 30m
maxRequestsPerConnection: 2147483646
maxRetries: 2147483646
tcp:
maxConnections: 2147483646
tcpKeepalive:
interval: 30s
time: 30s
loadBalancer:
simple: RANDOM
outlierDetection:
baseEjectionTime: 2m
consecutiveErrors: 2
interval: 5s
maxEjectionPercent: 50
minHealthPercent: 50
tls:
mode: ISTIO_MUTUAL
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: h2-consumer-gateway
namespace: istio-system
spec:
configPatches:
- applyTo: CLUSTER
match:
cluster:
service: app.consumer-gateway.svc.cluster.local
context: SIDECAR_OUTBOUND
patch:
operation: MERGE
value:
http2_protocol_options:
max_concurrent_streams: 47707
However neither of these things have changed in way over 6 months
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[x] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Not to break :-)
Steps to reproduce the bug
I've not managed to reproduce this
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.5.7
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
GKE 1.15
Activity
Stono commentedon Jul 5, 2020
I had a thought that maybe the DPE's (whatever caused those) in turn caused outlier detection to kick in, unfortunately we weren't scraping envoy metrics into prometheus so i don't have historical data, but one of the pods had:
For context there are 6
nginxand 6consumer-gatewaypods.However that doesn't explain why we were getting request failures as we have a max ejection % of 50? It’s like all hosts were evicted and the requests just stalled resulting in a 503.
I've raised #25219 which would be nice, ability to enable outlier logs on a pod level rather than globally
Stono commentedon Jul 5, 2020
Also it looks as if outlierDetection is not working as documented, we're getting evictions on 5xx codes despite explicit configuration to exclude them: #25220
Stono commentedon Jul 5, 2020
@mandarjog could this be some sort of unexpected interaction with http2+outlier?
@howardjohn and ideas? It was quite a nasty incident for us :(
rshriram commentedon Jul 6, 2020
For posterity, 1.5.7 was a security release and nothing more. The only commits were pertaining to the CVEs https://istio.io/latest/news/releases/1.5.x/announcing-1.5.7/
As you can see from the commit log https://github.com/istio/istio/commits/1.5.7 since 1.5.6 (june 15th), the only change has been to update the proxy SHA. If this issue is continuing to happen, can you try 1.5.6 proxy image (I know it has some CVEs mentioned above. So try it in some safer setup).
Stono commentedon Jul 6, 2020
We've only had one occurrence of this so far (thankfully) which unfortunately means that rolling back won't give a timely, deterministic data point. If we see it pop again, that will be what we try next.
howardjohn commentedon Jul 6, 2020
@lambdai I have never seen a DPE before.. any ideas what can cause this?
lambdai commentedon Jul 6, 2020
WRT
maxEjectionPercentnot working as expected: I recently fixed a bug in upstream envoy but not back ported. That config is activated only if the stats is enabled until the fix. Could you double check if the stats is enabled at your istio? That cluster stats is NOT enabled by defaultenvoyproxy/envoy#11629
lambdai commentedon Jul 6, 2020
Explained in #25220 that outlier detector sees a wider occurrence of 5xx. The metric shows outlier detector is triggered, likely the gateway error.
lambdai commentedon Jul 6, 2020
"DPE" is new to me, too. I need to dig
Stono commentedon Jul 6, 2020
Hey I've provided you my config dumps and metrics - can you confirm if i'm perhaps being hit by that bug @lambdai please and if so what the impact is as it isn't clear to me?
This is my stats config:
lambdai commentedon Jul 6, 2020
GHSA-8hf8-8gvw-ggvx is one of the vulnerability fix in 1.5.7
I am not fully sure your case falls into this bucket, but envoy is more aggressive in resetting streams and dest envoy says DPE (due to flood detection). Meanwhile the source envoy would interpret as 5XX(upstream failure)
lambdai commentedon Jul 6, 2020
istio/envoy@ea2d62e#diff-0cde9df72a69112b1d9af9cc414a107bR315
Is there any luck this counter is captured by prometheus?
Stono commentedon Jul 7, 2020
Yes that metric is there but its 0:
It's worth noting btw we only really see this under heavy load. Right now (about 50% load) we're hardly getting any outlier ejections active.
Unfortunately I was not able to enable the outlierLog because of another bug! #25269
I'm going to try rolling back to 1.5.6 to see what happens.
@lambdai I still don't understand if what that stats bug you fixed does, or if i'm impacted by it?
32 remaining items
Stono commentedon Dec 7, 2020
@howardjohn @mandarjog @lambdai we recently tried turning on http2 again on 1.7 (this issue was raised on 1.5) and can confirm after a few days (3) we started to see the same failures, and backed it out. I don't really have any more information other than what has been shared on this thread.
The extremely difficult thing here it is takes days to start manifesting and when it does it's only on a service that receives very high throughput - so capturing istio-proxy logs can be difficult.
Stono commentedon Dec 7, 2020
In fact one other piece of information I was able to glean from our edge logs was that the issues affected all different services routing through consumer-gateway.
So as a refresher,
consumer-gatewayis azuulproxy to a variety (20+) downstream apps. During the latest incident, as you can see here, all different paths (so subsequently all different services) started having errors. This points more towards an Istio level issue, rather than an application level issue like @lambdai pointed out (eg duplicated headers etc) as the impact was simply too broad (and all consumer-gateway does is blind reverse-proxy, and was not modified during this time).Stono commentedon Feb 28, 2021
As part of #31136 i've turned http2 on for one of our services again. This time we're on 1.8.3
Stono commentedon Oct 29, 2021
@howardjohn as you saw in slack we decided to make http2 the default again.
After a indeterminate amount of time, we start to observe
503UC.You can see from the trace it's effectively the first "hop" that fails:
With a
DPE:So whatever this issue is, it's still present
Stono commentedon Oct 29, 2021
Interestingly, it's the same service as last year (
ingress-nginx->consumer-gateway)mandarjog commentedon Oct 29, 2021
What are the response flag details ?
Is the following the correct flow?
Nginx-ingress + envoy —> (h2) —> envoy sidecar + zuul ( consumer gw)
This means the sidecar is encoding h2 which it cannot decode on the other side…
Stono commentedon Oct 29, 2021
What do you mean sorry? Is there some other info I can get for you?
Yes that's the correct flow
Yeah but why would that happen? And also, why would it just degrade without any releases or code changes, and be immediately fixed by toggling http2 on and off! This happened at 6am one morning, http2 had been enabled on
consumer-gatewayconfig quite some time before that.mandarjog commentedon Oct 29, 2021
https://istio.io/latest/docs/tasks/observability/logs/access-log/#default-access-log-format Look for %RESPONSE_CODE_DETAILS%
It is possible that this is data dependent. So some specific headers may bring out this behaviour in h2 processing .
@lambdai is there a way to dump full request contents when a DPE is detected? We may need a special istio-proxy build.
Stono commentedon Oct 29, 2021
So we don't have the access log enabled by default as its noisy and gives us little over what Jaeger does.
And restarting the workload will make the problem temporarily go away (so we can't add annotations).
Is there any way for me to enable access log, at run time?
Or perhaps an EnvoyFilter that would log out the request context when response 0DPE?
swarnar87 commentedon Apr 30, 2022
We have the exact same issue. Any way to log ingress request headers? This is the error in the logs:
[debug][http] [source/common/http/conn_manager_impl.cc:322] [C18109] dispatch error: http/1.1 protocol error: HPE_INVALID_HEADER_TOKEN