-
Notifications
You must be signed in to change notification settings - Fork 7.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
503 UC not being recorded in traces #50875
Comments
this should be report to envoy. |
Whilst on paper I agree, from a consumer experience perspective - I don't. Users of Istio will report issues to Istio. Some Istio users wouldn't even know there's Envoy under the hood as it's abstracted away. It's like my car, if my brakes aren't working - I'd report it to Tesla (the manufacturer of my car), not Bosch (the manufacturer of my brakes). |
To this date, this is still a problem :) |
Looking into it |
I'm going to try and recreate it with a sample app tomorrow, not sure how - but i'll give it a bash. Thinking i can write something that just tcp resets aggressively. Problem is with this it's only manifesting in production, and 1 in several million requests! :) |
I should add by the way that sometimes they are getting recorded; it seems to specifically be when we get the error |
@keithmattix - probably best to leave this one with me actually, it seems hit and miss for me so there's got to be a correlating factor that I'm missing. Will update here if i manage to reproduce with specifics |
Is this the right place to submit this?
Bug Description
It appears that traces aren't being recorded for certain scenarios that result in a 503. In this case; connection resets.
As you can see here, we've got a reasonable stream of 503UC's from
storage.googleapis.com
.I've come to investigate this and there are no traces being recorded for those errors. Only the 200/201's etc.
I enabled istio proxy debug logs and could see the error:
I'd be expecting to see traces for these failed requests in Jaeger? I do see other traces which result in a 503, so it appears to be 503UC specifically.
Version
Additional Information
These aren't being captured by any retries or anything like that, the app ends up seeing them (this is from app logs):
The text was updated successfully, but these errors were encountered: