-
-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nurseries cannot propagate StopAsyncIteration exceptions #393
Comments
Oh wow, this is a tricky bug you found. At first I thought it was #229, but it's actually different. (BTW, that concurrent zip is really cool! Even if it does seem to be uniquely suited to tickling this very weird bug :-).) Anyway, here's what's happening: A popular way to implement context managers is by writing a generator, and then decorating it with @acontextmanager
async def open_nursery():
# ... do setup stuff
try:
yield nursery
except BaseException as exc:
nursery._record_exception(exc)
await nursery._wait_for_all_tasks_to_exit()
raise nursery._combine_exceptions() Here, exactly one of the background tasks raised However, there's a problem: you're not allowed to raise Our async with trio.open_nursery() as nursery:
raise StopAsyncIteration But the only reason this works is because the So this really depends on a bunch of arcane details of how async context managers, async generators, and the nursery implementation all interact. Neat! But, uh... what do we do about it? I think the only option is to give up on using While you're waiting, though, here's a workaround: the problem is specifically with having a function called by class async_zip(object):
def __init__(self, *largs):
# I added the __aiter__() call here; it's nothing to do with this bug, but you should have it :-)
self.nexts = [obj.__aiter__().__anext__ for obj in largs]
# We put the "are we done?" state on the object so _accumulate can see it
self.done = False
async def _accumulate(self, f, items, i):
try:
items[i] = await f()
except StopAsyncIteration:
self.done = True
def __aiter__(self):
return self
async def __anext__(self):
if self.done:
# catch nasty users who try to re-iterate us after we said we were finished
# (another advantage of having it on the object)
raise StopAsyncIteration
nexts = self.nexts
items = [None, ] * len(nexts)
async with trio.open_nursery() as nursery:
for i, f in enumerate(nexts):
nursery.start_soon(self._accumulate, f, items, i)
if self.done:
raise StopAsyncIteration
# I also added the call to tuple() here because that's how zip() is supposed to work :-)
return tuple(items) Alternatively you could implement it as an async generator yourself (this is using the native async generator syntax, so requires Python 3.6+). Maybe this is clearer? I dunno: async def async_zip(*aiterables):
aiterators = [aiterable.__aiter__() for aiterable in aiterables]
done = False
while True:
items = [None] * len(aiterators)
async def fill_in(i):
try:
items[i] = await aiterators[i].__anext__()
except StopAsyncIteration:
nonlocal done
done = True
async with trio.open_nursery() as nursery:
for i in range(len(aiterators):
nursery.start_soon(fill_in, i)
if done:
break
yield tuple(items) |
acontextmanager is also a missing piece of python stdlib.
Are you considering to upstream it?
|
It's already upstream, as contextlib.asynccontextmanager. (I had nothing to
do with this :-).) The problem is that this won't actually ship until
Python 3.7 comes out, which is ~6 months away, and longer than that to
become the minimum supported version. I'll probably move trio's version
into async_generator for those who need it sooner. (If someone wants to
make a PR for this then please do!)
…On Jan 5, 2018 7:42 AM, "Imran Geriskovan" ***@***.***> wrote:
acontextmanager is also a missing piece of python stdlib.
Are you considering to upstream it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#393 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAlOaBrHiOscMgHjOW1HVNP64-NuVxY3ks5tHkLKgaJpZM4RUADX>
.
|
Ahh, thanks for explaining that (and little code fixes). Your first work-around will work nicely (I'm still on 3.5.2). |
I'm trying to implement an async version of
zip
using the following code, but I'm getting an error when one of the iterators passed toasync_zip
raises aStopAsyncIteration
before the others are finished. Normally I'd expect that I can catch it using the multierror or a try/except, however, instead I'm getting aRuntimeError
.I don't understand why this is happening and it's not clear if this is a bug in my code or in trio.
This prints
The text was updated successfully, but these errors were encountered: