Connection reset by peer when sending POST #4937
Comments
|
+1 same issue here |
|
test |
|
We are also experiencing a lot of errors (14%) on some servers. POST call is not retried by requests as it's not idempotent. After installing |
|
I can reproduce single |
|
I've got more info here.
When I moved initialization of session to method that is doing requests, connection errors disappeared. though that will not use session and will consume more resources. Shouldn't requests lib be aware of these closed connections and retry them automatically? |
|
Been experiencing this issue as well. In my case, session is being used and we are doing the actual request inside a coroutine ( |
|
I ran into this as well, and I can provide a way to reproduce 😄 To reproduceRun any of the notebooks in Current codeThe current code is in https://github.com/e-mission/e-mission-eval-public-data/blob/master/emeval/input/spec_details.py#L24 and uses the quickstart version without any sessions Workarounds triedI have tried multiple workarounds, none of which have worked:
At this point, I am tempted to give up on requests and drop down to urllib directly |
|
Also, I control both the client and the server, and I can confirm that the request that fails never makes it to the server. On the client: On the server, last call to |
|
ok, so after poking around a bit more, I actually see an error on the server This seems to imply that the client is dropping the connection to the server. But I am no longer sure that dropping directly to urllib is going to solve the problem. |
|
A workaround that did work was to simply catch the error and retry. I am still not sure whether the client or the server is dropping the connection, and why they are doing so, but hopefully this workaround helps somebody else. Note that both the request creation ( try:
response = requests.post(self.DATASTORE_URL+"/datastreams/find_entries/timestamp", json=post_msg)
print("response = %s" % response)
response.raise_for_status()
ret_list = response.json()["phone_data"]
except Exception as e:
# Hacky copy-paste of original code, TODO refactor into separate function
print("Got %s error %s, retrying" % (type(e).__name__, e))
time.sleep(10)
response = requests.post(self.DATASTORE_URL+"/datastreams/find_entries/timestamp", json=post_msg)
print("response = %s" % response)
response.raise_for_status()
ret_list = response.json()["phone_data"] |
|
Experiencing this same issue as well. It seems to be happening 50% of the time. |
|
+1, experiencing ~35% of the time and I am using urllib3 directly. |
|
Experiencing this issue as well. Made an exponential backoff to bypass it but, that seems somewhat hacky. |
|
I'm having the same issue. What version is it happening on? Im using 2.21.0 |
|
@LouisBellinger: Can you eleborate your exponential backoff mechanism to bypass the issue? |
|
Hi to everyone, Any update to this issue? I'm having the same problem. Do you known any hotfix? |
|
Is there a workaround? |
|
for me, a single retry has worked pretty reliably: For exponential backoff, you would just catch the exceptions multiple times, sleeping longer every time. |
|
Any solutions? |
|
Hello... The Persistent Connectionts features are not handling the client side when it requested a Connection Keep Alive, because at lease for Linux based OS the TCP Keep Alive Feature is not enabled, or when Enabled it is set to 2 start working after 2 hours, so I found that the problem iis because the OS is not handling the flo control correctly you can fix it in some ways... For Python in Linux import requests now you can Instanciate your Sessions or Requests.. Without TCP Keep Alive With TCP Keep Alive
If the client reach the unhandled session timeout by the server...now the server will close the connection , releasing the connection back to the pool insteadd of dropoing it....
There are some other ways to handle it.. depending on your OS and in your needs... |
|
I found this but I dont how to handle setsockopt and requests... maybe you know. and will fix ti for all OSs |
|
I was also having this issue (and not just on POST requests) while running stress tests of services using locust, which uses requests. The very good investigation and fix provided by @jakermx really did it for me as well: import socket
from urllib3.connection import HTTPConnection
# ...
HTTPConnection.default_socket_options = (
HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45),
(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10),
(socket.SOL_TCP, socket.TCP_KEEPCNT, 6)
]
)Insert that code just somewhere that gets loaded by your application/script and it will change the This isn't a bug in requests nor urllib3, since both are offering solutions:
Also, urllib3 makes it pretty clear that especially in this case, there is a simple detection issue for automatic handling: |
|
By defautl, OS TCP/IP stack defines that the app will stack sending TCP Keep-Alives packes after 2 hours of idle activity, this will only work on OOTB Web Servers Installations, but, if your Content provider set lower limits, the TCP L5/L4 Connection willbe dropped off before 2 hours....and when your app try to used the "persistent" connection it will fail due Connection Reset by RemotePeer...caus it is not a valid connection, setting the tcp options will avoind someerrorsm but not all....not matter is you set the connection keep alive header at L7-L4....if the OS dont send KA packets at TCP layer....the only header that will avoid this conditions is Connection:Close this is a app, http friendly API, url m api and IETFissue |
|
@jakermx you are my hero. Thank you for solution. |
|
there is no heros pal,,,,just a great community workforce |
|
There are several issues that have been tacked onto the original problem here because people are conflating multiple exceptions. I believe @jakermx has explained what is happening at the TCP level and the urllib3 issue #944 discussing that we're not able to distinguish between a dropped connection and RST has been linked. I'm going to resolve this as there isn't more information to provide at this point. |
Client code that uses Requests module to send data via HTTP POST encounters a
ConnectionResetError. The entire operation (composed of multiple POST requests to a small set of service endpoints) can sometimes succeed, but most of the time, it fails with this error.Expected Result
Operation succeeds (or fails) without connection issues.
Actual Result
The operation fails with
ConnectionResetError.Additional Information
It's a little difficult to provide basic reproduction for the issue as we're running into this problem with a test payload that's specific to our system. The server (peer) is a Java application that's configured to terminate/reset connection after a given time of no use (idle). The client sends multiple one-off POST requests, but it seems like internally, the connections are being reused similar to issue #4506, and the operation eventually runs into a connection that's already been reset and raises the error. However, unlike #4506, we are not using Sessions.
Here are some tracebacks that could hopefully describe the problem better:
System Information
The text was updated successfully, but these errors were encountered: