Implemenation of the PEP 492 - Coroutines with async and await syntax

[vstinner]

BPO	24017
Nosy	@gvanrossum, @ncoghlan, @scoder, @vstinner, @asvetlov, @agronholm, @1st1
Dependencies	bpo-2292: Missing *-unpacking generalizations bpo-22906: PEP 479: Change StopIteration handling inside generators bpo-24018: add a Generator ABC bpo-24180: PEP 492: Documentation bpo-24184: PEP 492: Add AsyncIterator and AsyncIterable to collections.abc bpo-24315: collections.abc: Coroutine should be derived from Awaitable bpo-24316: Fix types.coroutine to accept objects from Cython bpo-24400: Awaitable ABC incompatible with functools.singledispatch
Files	await_01.patch await_02.patch await_03.patch await_04.patch await_05.patch await_06.patch with.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/1st1'
closed_at = <Date 2015-05-13.19:46:15.300>
created_at = <Date 2015-04-20.19:33:55.846>
labels = ['interpreter-core', 'type-feature', 'release-blocker']
title = 'Implemenation of the PEP 492 - Coroutines with async and await syntax'
updated_at = <Date 2015-06-20.19:39:53.655>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-06-20.19:39:53.655>
actor = 'yselivanov'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2015-05-13.19:46:15.300>
closer = 'yselivanov'
components = ['Interpreter Core']
creation = <Date 2015-04-20.19:33:55.846>
creator = 'vstinner'
dependencies = ['2292', '22906', '24018', '24180', '24184', '24315', '24316', '24400']
files = ['39148', '39155', '39314', '39325', '39337', '39344', '39354']
hgrepos = []
issue_num = 24017
keywords = ['patch']
message_count = 69.0
messages = ['241678', '241679', '241687', '241697', '242056', '242064', '242086', '242304', '242326', '242627', '242731', '242779', '242793', '242848', '242860', '242862', '242877', '242912', '242927', '242928', '242931', '242935', '242936', '242937', '242938', '242979', '242982', '242984', '243031', '243033', '243035', '243036', '243037', '243038', '243041', '243042', '243117', '243118', '243119', '243759', '244175', '244252', '244257', '244258', '244261', '244264', '244268', '244269', '244273', '244278', '244280', '244281', '244303', '244307', '244319', '244322', '244374', '244384', '244385', '244567', '244568', '244596', '244759', '244761', '244799', '244818', '244826', '244831', '245075']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'ncoghlan', 'scoder', 'vstinner', 'asvetlov', 'alex.gronholm', 'python-dev', 'yselivanov']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24017'
versions = ['Python 3.5']

[1st1]

Here's the first patch (git diff master..await). Should be easier to review and work from there.

[1st1]

Attaching a patch generated with mercurial

[vstinner]

Does the implementation depends on the implementation of the PEP-479? (issue bpo-22906)

Attaching a patch generated with mercurial

Next time, if possible, try to skip generated files. Maybe write a script for that, but sorry I don't know how :-(

[1st1]

Attaching a revised patch (all Victor's comments but asyncio changes)

[scoder]

Could we have type slots for the new special methods? Otherwise, implementing the protocol in C would be fairly inefficient, especially for async iteration.

I'm asking because Cython's generator type is not Python's generator type, but implementing the rest of the proposed protocol doesn't seem to be all that difficult.

[1st1]

Could we have type slots for the new special methods? Otherwise, implementing the protocol in C would be fairly inefficient, especially for async iteration.

I don't think it's necessary to have slots for __aiter__, __anext__, __aenter__ and __aexit__. Async iteration will never be as fast as regular iteration, and there is plenty overhead in it. And we don't need slots for async context managers as we don't have slots for regular ones.

What might be a good idea is to add a slot for __await__ method -- tp_await. This will allow to implement Futures in C efficiently. I'd rename 'tp_reserved' for this purpose.

Victor, your thoughts?

[1st1]

In fact I will likely add tp_await in the next PEP iteration. I need it to implement another feature.

[scoder]

I don't think it's necessary to have slots for __aiter__, __anext__, __aenter__ and __aexit__. Async iteration will never be as fast as regular iteration, and there is plenty overhead in it.

You seem to be assuming that the outer loop is the asyncio I/O loop. That might not be the case. It might be a thread-pool executor, it might be an in-memory task switcher, or it might just be something passing on items from a list. At least "__anext__" is worth being fast, IMHO.

Also note that one advantage of slots is that the user can cache their value to keep calling the same C function multiple times. That is not the case for a Python method, which can easily be replaced. Some iterators do that with their __next__ method, and it's perfectly valid. Having to look up the Python method on each iteration and calling through it sounds like unnecessary overhead.

[gvanrossum]

I think we can continue this discussion *after* the PEP's been accepted.

[1st1]

I'll upload the most recent patch soon.

[1st1]

Third patch attached. Victor, it would be great if you can review it!

[1st1]

Another iteration:

• support of new syntax in lib2to3
• collections.abc.Awaitable