Nurseries cannot propagate StopAsyncIteration exceptions #393
Comments
|
Oh wow, this is a tricky bug you found. At first I thought it was #229, but it's actually different. (BTW, that concurrent zip is really cool! Even if it does seem to be uniquely suited to tickling this very weird bug :-).) Anyway, here's what's happening: A popular way to implement context managers is by writing a generator, and then decorating it with @acontextmanager
async def open_nursery():
# ... do setup stuff
try:
yield nursery
except BaseException as exc:
nursery._record_exception(exc)
await nursery._wait_for_all_tasks_to_exit()
raise nursery._combine_exceptions()Here, exactly one of the background tasks raised However, there's a problem: you're not allowed to raise Our async with trio.open_nursery() as nursery:
raise StopAsyncIterationBut the only reason this works is because the So this really depends on a bunch of arcane details of how async context managers, async generators, and the nursery implementation all interact. Neat! But, uh... what do we do about it? I think the only option is to give up on using While you're waiting, though, here's a workaround: the problem is specifically with having a function called by class async_zip(object):
def __init__(self, *largs):
# I added the __aiter__() call here; it's nothing to do with this bug, but you should have it :-)
self.nexts = [obj.__aiter__().__anext__ for obj in largs]
# We put the "are we done?" state on the object so _accumulate can see it
self.done = False
async def _accumulate(self, f, items, i):
try:
items[i] = await f()
except StopAsyncIteration:
self.done = True
def __aiter__(self):
return self
async def __anext__(self):
if self.done:
# catch nasty users who try to re-iterate us after we said we were finished
# (another advantage of having it on the object)
raise StopAsyncIteration
nexts = self.nexts
items = [None, ] * len(nexts)
async with trio.open_nursery() as nursery:
for i, f in enumerate(nexts):
nursery.start_soon(self._accumulate, f, items, i)
if self.done:
raise StopAsyncIteration
# I also added the call to tuple() here because that's how zip() is supposed to work :-)
return tuple(items)Alternatively you could implement it as an async generator yourself (this is using the native async generator syntax, so requires Python 3.6+). Maybe this is clearer? I dunno: async def async_zip(*aiterables):
aiterators = [aiterable.__aiter__() for aiterable in aiterables]
done = False
while True:
items = [None] * len(aiterators)
async def fill_in(i):
try:
items[i] = await aiterators[i].__anext__()
except StopAsyncIteration:
nonlocal done
done = True
async with trio.open_nursery() as nursery:
for i in range(len(aiterators):
nursery.start_soon(fill_in, i)
if done:
break
yield tuple(items) |
njsmith
changed the title
|
acontextmanager is also a missing piece of python stdlib.
Are you considering to upstream it?
|
|
It's already upstream, as contextlib.asynccontextmanager. (I had nothing to
do with this :-).) The problem is that this won't actually ship until
Python 3.7 comes out, which is ~6 months away, and longer than that to
become the minimum supported version. I'll probably move trio's version
into async_generator for those who need it sooner. (If someone wants to
make a PR for this then please do!)
…On Jan 5, 2018 7:42 AM, "Imran Geriskovan" ***@***.***> wrote:
acontextmanager is also a missing piece of python stdlib.
Are you considering to upstream it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#393 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAlOaBrHiOscMgHjOW1HVNP64-NuVxY3ks5tHkLKgaJpZM4RUADX>
.
|
|
Ahh, thanks for explaining that (and little code fixes). Your first work-around will work nicely (I'm still on 3.5.2). |
I'm trying to implement an async version of
zipusing the following code, but I'm getting an error when one of the iterators passed toasync_zipraises aStopAsyncIterationbefore the others are finished. Normally I'd expect that I can catch it using the multierror or a try/except, however, instead I'm getting aRuntimeError.I don't understand why this is happening and it's not clear if this is a bug in my code or in trio.
This prints
The text was updated successfully, but these errors were encountered: