asyncio CancelledError raised from SSLWantReadError handler in _call_sslobject_method #534
Comments
|
This looks like:
Am I mistaken? |
|
I read it the same, but I'm at loss as to why "The read operation is cancelled"... I'm new to Python's async stack, so I apologize if anyio is a wrong place for this issue, but I don't know how to start debugging a CancelledError thrown from underneath 4 levels of networking libraries :) Is it generally an expected behavior that this low-level error can be thrown from anyio's code? Could anyio instead raise a higher-level exception with a more specific cause as to why the stream ended up in an invalid state? |
|
Well, apparently … … this timeout expired. Thus anyio cancelled the operation, which is done by injecting a Yes, there's a whole bunch of unexperienced-programmer-confusing tracebacks attached to what should be a straightforward and "simple" exception, but ultimately there doesn't seem to be any real problem here. |
|
I think the confusion is that you thought |
No, that part I actually understood and mentioned in the question, but I would've never guessed that |
Well, it doesn't. Not really. It represents, surprise, the fact that part of your code got cancelled. That might happen because of a timeout, or an exception in some other unrelated part of your code that propagated out of a taskgroup (which cancels all the other tasks in it, otherwise we'd violate Structured Concurrency guarantees), or …. |
|
I guess what confused me the most was the inverse logic of the backtrace, which, due to Python's message ( |
|
We often see read timeouts happening with big company APIs. |
This is my life right now, someone please kill me |
|
Are you still having issues? Have you tried v4.0.0rc1 yet? |
|
Oh uh hey there! Thanks for the quick reply. I'm on |
|
If it's caused by an AnyIO cancel scope, the cancel message will show which cancel scope it is (by object ID) in v4.0.0rc1 (which btw appears to be already compatible with |
|
We had a lengthy conversation on the Trio Gitter room about tracking cancellation origin just a few days ago. Some interesting ideas came up. It will require some effort from both projects (AnyIO and Trio) in order to work. |
|
Sweet, okay I'll give that a try. Yeah I know what you mean, I'm new to async and tracking down stray CancelledErrors has been a real mindfuck. |
|
… almost as bad as figuring out which of N shielded cancel scopes (those are pretty much required for async code in |
|
Not to mention, it could also be a stray thread that's causing it. The 4.0 series should decrease the number of these headaches a bit by the means of bugfixes. |
|
Yikes. The worst part of all this is I'm not even sure which gods we should be praying to. Who's fault is it? Who's gonna save us? CPython? AnyIO? Mojo? ¯_(ツ)_/¯ |
|
Anyway @agronholm for the sake of record, this is the traceback I'm dealing with. It appears unexpectedly out of httpx about 1 out of every 1000 web requests. It is the same every time. I'd love to be able to handle only this specific CancelledError as a temporary solution, but there doesn't seem to be a way to determine which task was cancelled or where. |
|
Concurrency in programming is hard. We're trying to make it less painful (structured concurrency helps a lot), but we still have ways to go. As far as your traceback goes, it's hard to say where this cancellation comes from, but it doesn't seem to come from AnyIO, as the exception would not fall out of a task. If you want to be sure, try with AnyIO 4.0 (you should see a message associated with the exception if, by some chance, an AnyIO exception did in fact fall out of a task). It's a good idea to switch anyway given the myriad cancellation related fixes that went into the release. Best of luck! |
|
This looks suspiciously like a bug I've found. If your app uses memory object streams, 4.0 should fix this. |
|
Btw, I was able to confirm this was an AnyIO bug, and that it's been fixed in 4.0.0rc1. This was the code that triggered the import httpx
import random
import asyncio
## STEP 1:
## python -m http.server 5555
async def main():
async def request():
transport = httpx.AsyncHTTPTransport(retries=5) # <-----------------------
async with httpx.AsyncClient(transport=transport, timeout=random.random() / 10) as client:
try:
return await client.get("http://127.0.0.1:5555")
except httpx.TimeoutException:
print("_", end="", flush=True)
while 1:
tasks = [asyncio.create_task(request()) for _ in range(100)]
await asyncio.gather(*tasks)
asyncio.run(main())@agronholm thanks again for your help. |
|
You really should not use |
|
Also why are you creating an HTTP client for every single request? That's very inefficient. |
In an async Python application, I get the following traceback:
As far as I understand,
SSLWantReadErrorshould be handled inside anyio, but theCancelledErrorthat gets thrown by the exception handler prevents that from happening. Also, it seems like there is some logic in place to catchCancelledErrorin other places, so I thought this might be a bug in anyio.Relevant library versions:
The text was updated successfully, but these errors were encountered: