avoid explicit generator type check in asyncio

[scoder]

BPO	24004
Nosy	@gvanrossum, @scoder, @vstinner, @vadmium, @1st1
Files	coroutine.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-31.01:06:23.847>
created_at = <Date 2015-04-19.10:21:02.253>
labels = ['type-bug', 'expert-asyncio']
title = 'avoid explicit generator type check in asyncio'
updated_at = <Date 2015-05-31.15:29:12.912>
user = 'https://github.com/scoder'

bugs.python.org fields:

activity = <Date 2015-05-31.15:29:12.912>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-31.01:06:23.847>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2015-04-19.10:21:02.253>
creator = 'scoder'
dependencies = []
files = ['39569']
hgrepos = []
issue_num = 24004
keywords = ['patch']
message_count = 18.0
messages = ['241506', '241595', '241609', '241676', '243782', '244453', '244459', '244474', '244479', '244482', '244488', '244497', '244498', '244507', '244508', '244524', '244543', '244544']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'scoder', 'vstinner', 'python-dev', 'martin.panter', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24004'
versions = ['Python 3.5']

[scoder]

asyncio/coroutines.py contains this code:

"""
_COROUTINE_TYPES = (types.GeneratorType, CoroWrapper)

def iscoroutine(obj):
    """Return True if obj is a coroutine object."""
    return isinstance(obj, _COROUTINE_TYPES)
"""

In other places, it uses inspect.isgenerator() to do the same thing. This is inconsistent and should be changed. Code that wants to patch inspect.isgenerator() in order to support other generator types shouldn't also have to patch asyncio directly, and code that wants to support other generator types in asyncio shouldn't have to patch both places either.

[gvanrossum]

Uh, wait. Who's patching anything? That breaks the warranty.

[scoder]

I was (silently) hoping that this patching would eventually not be necessary anymore because the one place (currently inspect.isgenerator()) would be adapted to check for the generator protocol rather than the generator type. But that was going to go into a separate ticket. :)

I'm not sure inspect.isgenerator() is the right place, though, as the module tends to deal with types, not protocols. I think the right fix overall would be a Generator ABC class that defines the protocol.

What Cython does now in order to make asyncio work with Cython compiled generators is this:

https://github.com/cython/cython/blob/4af42443bd37f6207143f8870904f780a65bb3e3/Cython/Utility/Generator.c#L824

https://github.com/cython/cython/blob/4af42443bd37f6207143f8870904f780a65bb3e3/Cython/Utility/Generator.c#L906

Aweful, but currently required due to the type check, which prevents objects that implement the generator protocol from being handled correctly by asyncio.

[scoder]

I created bpo-24018 for adding a Generator ABC to collections.abc.

[1st1]

This issue was resolved in python/asyncio@3a09a93.

[scoder]

I found one more place in asyncio.coroutines, around line 190 in the coroutine() decorator:

    if inspect.isgeneratorfunction(func):
        coro = func
    else:
        @functools.wraps(func)
        def coro(*args, **kw):
            res = func(*args, **kw)
            if isinstance(res, futures.Future) or inspect.isgenerator(res):
                res = yield from res
            return res

The wrapped isgenerator() check should be replaced by an Awaitable ABC check, e.g. this works for me:

"""
res = func(*args, **kw)
if isinstance(res, futures.Future) or inspect.isgenerator(res):
res = yield from res
+ else:
+ await_res = getattr(res, '__await__', None)
+ if await_res is not None:
+ res = yield from await_res()
return res
"""

[1st1]

What inspect.isgeneratorfunction(func) returns for Cython generators?

[scoder]

Cython functions that return a generator aren't special in any way, so it's actually correct that isgeneratorfunction() always returns False. I mean, I could set the CO_COROUTINE flag on the code object that we fake, but that wouldn't help much as Cython's functions are not Python functions, so isfunction() will also return False for them and no-one would look at the flag in the first place. And finally, a Cython generator is not a Python generator (with byte code position etc.), so isgenerator() also correctly returns False.

At least iscoroutine() and iswaitable() will return True for Cython Coroutines now.

[scoder]

Hmm, I just noticed that this only started failing when I disabled the asyncio module patching for 3.5beta1+, assuming that it now all works. A side effect of that was that also the inspect module is no longer patched into returning True from isgenerator() for Cython generators. While there might be code that fails when it tries to inspect Cython generators as Python generators (as the first don't have byte code), I think it's still the easiest fix to simply keep the inspect patching. That would then trigger the right code path here without changes in asyncio.

[scoder]

... except that the returned object may not actually be a Cython generator but a GeneratorWrapper, now that the patch from bpo-24316 is in place. :)

Then I guess we're back to the point where duck-typing and calling __await__() is a good idea.

[1st1]

Stefan,

Then I guess we're back to the point where duck-typing and calling __await__() is a good idea.

Please test the attached patch. I agree this is a good idea.