-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Envoy proxy not working for HTTP/1.1 requests which are multi-part form uploads #13240
Comments
Please refer to this link for debug logs: istio/istio#27423 (comment) |
I have looked at the log and it seems the the upstream server has closed connection about 500ms after it was established and request was sent. You may want to look at the logs on the upstream service to see if there was a reason for closing the connection.
|
Does Envoy enforce a 500ms timeout by default? The upstream service is known to take about 5000ms since this is a data upload. When I look at the application logs, it does look like the request is received, but it eventually gets flooded with the same requests, because of envoy's retry policy and eventually returns a |
Would trace logs give us more info here? I usually shy away trace logs because it ends up emitting way too much info |
I does not look like Envoy's timeout, since it is the remote (from Envoy's perspective) end that closed the connection. Are you suggesting to enable trace logging on Envoy? I doubt it would yield more information. I think the key piece is why the remove end had closed the connection. |
@yanavlasov We figured out the issue. It was caused by the Envoy proxy OOMing on the server's end. Looks like the server's envoy needed more memory to process the multipart request. I don't claim to be any envoy expert, but what I don't get is that the client-side envoy faces no memory issues while trying to send this multi-part request. Isn't the volume of data received same as that of what is sent? |
Are both Envoys configured the same? Especially buffer sizes? If the buffers are too large it is possible that one Envoy quickly pushes too much data into its peer causing it to run out of memory. |
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions. |
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions. |
Can we please reopen this? I have the similar issue with And I tested by curl with a tiny 204 bytes file Test detail:
|
Hitting this issue, anyone know of a workaround? |
When attempting to upload a file, I receive a |
Description:
We are facing issues when two services (which are sidecar'ed with envoy proxies) communicate over HTTP and attempt a multi-part POST requests. Please note that other POST and GET requests work fine with no issues.
We get the generic envoy 503 error:
upstream connect error or disconnect/reset before headers
. I am unable to understand what exactly went wrong from this error.These envoy proxies are managed through Istio and there have been multiple issues posted highlighting this issue:
istio/istio#27423
istio/istio#27373
istio/istio#25734
Any insights on this would be appreciated. What do we look for? What most likely could have gone wrong as far as envoy proxy configuration is concerned?
The text was updated successfully, but these errors were encountered: