"Awaitable" is insufficiently expressive, as one may await something other than Futures

[glyph]

In the wild, today, if you await something, there are several potentially valid things to bubble back out from the innermost await

• an asyncio.Future
• a curio "trap" (which I gather is defined as something like Tuple[curio.traps.Traps, Any, ...])
• a Trio trap, (which I gather is defined as something like Iterable[WaitTaskRescheduled] although I haven't quite found the bottom of that stack)
• a Deferred

Not all of these are mutually compatible. However, Awaitable is parameterized only on one type - the type of its result. This results in code like this:

def some_deferred() -> Awaitable[int]: ...
def some_future() -> Awaitable[int]: ...

async def x() -> Awaitable[List[int]]:
    y = []
    y.append(await some_deferred())
    y.append(await some_future())
    return y

type-checking even though it should fail.

I'd like to be able to express this as

def some_deferred() -> Awaitable[int, Deferred]: ...
def some_future() -> Awaitable[int]: ... # presumably Future is the default

async def x() -> Awaitable[List[int]]:
    y = []
    y.append(await some_deferred()) # <-- automatically specialize to `Awaitable[List[int], Deferred]` here, if possible?
    y.append(await some_future()) # <-- type error!
    return y

This is made slightly more complex by the fact that some awaitables are mutually intelligible - for example, anything that awaits a Deferred should eventually be able to await a Future directly, once we've made a few more changes to Twisted. We could also do this in the other direction if __await__ were changed to pass the loop along. Whereas, some of these types are incompatible by design; my understanding is that await implies a completely different, and somewhat higher-level, scheduling protocol in Trio, for example.

[ilevkivskyi]

This looks like something quite tricky. After thinking for some time I didn't find a solution that wouldn't require a lot of special casing in type checkers.

(Also note that by agreement async defs are annotated without wrapping return type in Awaitable, see PEP 484.)

[JelleZijlstra]

I'm actually not sure I understand the example. As Ivan says, the type annotation on x seems wrong—it should be -> List[int].

But I also don't see why you want a TypeError in your second code block. If you run await some_deferred() or await some_future(), either way you will get an int, so you are still simply making a list of ints. some_deferred() itself may return a Deferred[int], but that can just be implemented by making Deferred a subclass of Awaitable.

[gvanrossum]

It seems Glyph would like mypy to tell him that he's (incorrectly, presumably) mixing await <future> and await <deferred> in the same code. The problem is that there's nothing in the signature of the async function that tells us which it should be -- the class Awaitable doesn't actually occur there, according to PEP 484. So I'm with Ivan -- it's unclear how to proceed. Some fundamental change in the type system would be needed. Or we would need to reconsider the choice in PEP 484 for annotating async functions -- but even so we would need to design its replacement, and generic classes currently don't support a variable number of arguments, so it would become a complex change proposal there.

…

On Fri, Mar 9, 2018 at 3:57 PM, Jelle Zijlstra ***@***.***> wrote: I'm actually not sure I understand the example. As Ivan says, the type annotation on x seems wrong—it should be -> List[int]. But I also don't see why you want a TypeError in your second code block. If you run await some_deferred() or await some_future(), either way you will get an int, so you are still simply making a list of ints. some_deferred() itself may return a Deferred[int], but that can just be implemented by making Deferred a subclass of Awaitable. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#542 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMl0I-85sbYKMYigcRmE3YcQaK_uzks5tcxb3gaJpZM4SjKi_> .

-- --Guido van Rossum (python.org/~guido)

[glyph]

@JelleZijlstra I think you do have a good point, that Deferred should be a subclass of Awaitable and appropriately generic.

One really nasty thing about the way Deferreds are constructed, unfortunately is that the stubs want to look something like this:

class Deferred(Generic[T]):
    def addCallback(self, cb: Callable[[T], T2], ...) -> Deferred[T2]: ...

Except, uh, addCallback returns self and its type changes at runtime.

I think maybe just pretending that it doesn't is the best way forward for type annotations in Twisted but I do wish there were some way to express this.

[glyph]

It seems Glyph would like mypy to tell him that he's (incorrectly, presumably) mixing await <future> and await <deferred> in the same code.

Yep. Fixing this is pretty easy even now: await deferred.asFuture(loop) or await Deferred.fromFuture(future) depending on which way you're going. But you do still need to be explicit about it.