Write Error on reusing potentially closed connection #44
Comments
|
I'm going to close this one off given the ancedotal encode/httpx#841 (comment) "I just tested the original gist against httpx==0.13.0.dev2 and got the same result for http1.1 and http2." If there's something reproducible here or any anecdotal reports of similar connection closing then we can reopen it, but given that it's just "the server closed the connection" there's not a lot here for us to act on. |
|
I have a service that after upgrading httpx to 0.13 started receiving the same error, and rolling back to 0.12 seems to resolve the issue. |
|
I don't know what details I can give here, other that:
|
|
Stack trace: |
|
TRACE: |
|
What it seems: after a while, the connection closes, but httpcore does not now that, attempts to write the request and throws an error. Does httpx 0.12 reuses http 1.1 connections? I'm seeing in the trace that it is always using a new connection in http 1.1, as 0.13.2 reuses a connection, which can be the cause. |
|
Hey @victoraugustolls, thanks for the detailed report. Reopening and temptatively renaming this issue. |
yeraydiazdiaz
changed the title
|
Thanks for the info - Is there a minimal reproducible example we can use to dig into this? |
httpcore does reuse HTTP/1.1 connections if a previous request has been closed (the connection's state is
There's a specific check in the connection pool for closed connections, I remember @florimondmanca working on the asyncio version and discussing it with core developers so I'd be surprised if that's not working as expected but of course you never know. It'd be also interesting to know how many concurrent requests are you attempting when the error occurs. We could also do with more trace logging on the |
In my case was a specific host because it was my only test. I will try to update my minimal case repo to reflex this issue!
Only 1, local server with 1 request at a time using Postman. My tests were 1 request everything works. 5 minutes later I do another request and it fails when reusing the connection. |
|
Okay needs looking into then. The connection ought to have been expired, as we have a keep-alive timeout, and close connections once they pass that mark. |
|
Could the host in question be closing the connection before the returned keep alive duration, causing this issue? |
I ask this because the host is a FastAPI server, running with uvicorn worker o gunicorn, with keep alive settings = 120. After exactly 121 seconds I receive the described error. I changed the setting to 240 and I still receive the error after 121 seconds. Response headers: |
|
So, on 0.13 if I force http/2.0 enabling it on the host, I have the same error after 121s. The trace: Traceback: |
|
Trace on httpx=0.12: Wonder why does it start with closing the connection. |
If you're talking about The default behaviour is to check if the connection has been dropped on the next request and create a new one if so, which seems to works fine on my tests. On my Mac, using the example HTTP/1.1 Uvicorn app with TLS, a keepalive timeout of 5 seconds, the enhanced trace logs in #101 and this simple script: import asyncio
import httpx
async def main() -> None:
async with httpx.AsyncClient(verify=False) as client:
await request(client)
await asyncio.sleep(60)
await request(client)
async def request(client: httpx.AsyncClient, url: str = None) -> None:
url = url or "https://localhost:8000"
response = await client.get(url)
print("==== Response data: ====\n")
print(response.http_version)
print(response.status_code)
print(response.headers)
print("\n==== Response body: ====\n")
print(response.text[:100], "...\n\n")
if __name__ == "__main__":
asyncio.run(main())No exceptions are raised and I get the following output: Note how the connection is not closed after reading the body and when the next request is sent we see:
Proving it detected the connection being closed and proceeded to create a new one. I guess it's possible our detection of dropped connections fails on your architecture, can you detail what you are using? |
|
Oh right sure... asyncio's dropped connection is less robust than the standard sync case, so we introduced a fairly short keep-alive period on the httpx side, which in practise means that it's not an issue since we'll expire the connections anyways... https://github.com/encode/httpx/blob/0.12.1/httpx/_dispatch/connection_pool.py#L89 However, when we switched over to httpcore in 0.13, we switched keepalive_expiry=5.0 when instantiating the connection pool / proxy classes from httpx... https://github.com/encode/httpx/blob/093cb4250006dc0b73af986f7e6537b8c4395dcf/httpx/_client.py#L516
This right thing for use to do first is switch those However it'd also be really helpful to have another dig into the "is the connection still alive" detection, and see if there's anything we can do to improve the robustness there. What'd be helpful for that would be having a minimal reproducible example, so...
async def app(scope, receive, send):
assert scope['type'] == 'http'
await send({
'type': 'http.response.start',
'status': 200,
'headers': [
[b'content-type', b'text/plain'],
],
})
await send({
'type': 'http.response.body',
'body': b'Hello, world!',
})$ venv/bin/uvicorn example:appIf not, then what extra config, or setup do you need to introduce before being able to reproduce?
import asyncio
import httpx
async def main():
async with httpx.AsyncClient() as client:
r = await client.get('http://127.0.0.1:8000/')
print(r.status_code)
print(r.text)
await asyncio.sleep(10)
r = await client.get('http://127.0.0.1:8000/')
print(r.status_code)
print(r.text)
asyncio.run(main())
|
|
Resolved in httpx 0.13.3, but leaving this open ATM, as it would still be good to dig into making our “connection is still open” is as robust as possible. |
My architecture is the host is a FastAPI service running behind gunicorn 20.0.4 inside a docker container in Azure App Service. The client follows the same architecture. |
|
@tomchristie I will check your comments during the weekend and come with detailed comments after, if that's ok. I will also apply #101 locally into httpx=0.13.2 to help with the trace. |
This snippet is enough to replicate, using sleep of 130 seconds.
No, only asyncio version of the snippet. I tried the 3 versions 5 times. Running against the server while running it in Azure App Service and running locally. Locally I do not have this error, but it is an http app while on Azure it is https, which can be part of the cause.
Locally Arch Linux (running code inside docker). Remotely Azure App Service. |
Ok, so running against https locally and http remotely still gives an error with the api running on Azure. So I guess the option here is to deploy a mock service on azure so I can send the url here? (verify=False gives same result, error while running against Azure) |
|
I'm a bit confused here, let's see if we can summarize the situation:
Is that correct? Could you share details on the client Docker image? |
|
Closing as stale. |
|
@tomchristie @yeraydiazdiaz I have missed the notification about you comments, I'm deeply sorry. After encode/httpx#1005 the error stopped happening! So closing this is fine 😄 |
With a service deployed with httpx version with httpcore interface, I started receiving the following error:
<class 'httpcore._exceptions.WriteError'>: [Errno 104] Connection reset by peerThe only problem is that this is when connecting to a service that others applications have no problems. The connection is (should be) HTTP/1.1 in this case.
The text was updated successfully, but these errors were encountered: