async generator receives wrong value when shared between coroutines

[dimaqq]

BPO	30773
Nosy	@njsmith, @cjerdonek, @dimaqq, @1st1, @emilyemorehouse, @jancespivo, @tirkarthi
PRs	bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel #7468 [3.8] bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (GH-7468) #16486

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/1st1'
closed_at = <Date 2019-09-30.06:25:31.467>
created_at = <Date 2017-06-26.18:55:33.730>
labels = ['interpreter-core', '3.8', 'type-bug', '3.7']
title = 'async generator receives wrong value when shared between coroutines'
updated_at = <Date 2019-09-30.06:25:31.466>
user = 'https://github.com/dimaqq'

bugs.python.org fields:

activity = <Date 2019-09-30.06:25:31.466>
actor = 'yselivanov'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2019-09-30.06:25:31.467>
closer = 'yselivanov'
components = ['Interpreter Core']
creation = <Date 2017-06-26.18:55:33.730>
creator = 'Dima.Tisnek'
dependencies = []
files = []
hgrepos = []
issue_num = 30773
keywords = ['patch']
message_count = 9.0
messages = ['296931', '296932', '297022', '317563', '317603', '317605', '317607', '353543', '353544']
nosy_count = 7.0
nosy_names = ['njs', 'chris.jerdonek', 'Dima.Tisnek', 'yselivanov', 'emilyemorehouse', 'jan.cespivo', 'xtreak']
pr_nums = ['7468', '16486']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30773'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

Linked PRs

• gh-74956: Add tests for async generator behaviour #111212

[dimaqq]

MRE

import asyncio


async def generator():
    while True:
        x = yield 42
        print("received", x)
        await asyncio.sleep(0.1)


async def user(name, g):
    print("sending", name)
    await g.asend(name)


async def helper():
    g = generator()
    await g.asend(None)

    await asyncio.gather(*(user(f"user-{x}", g) for x in range(3)))


if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(helper())

Produces output:

sending user-0
received user-0
sending user-1
sending user-2
received None
received None

Expected output (some variance allowed):

sending user-0
received user-0
sending user-1
sending user-2
received user-1
received user-2

Initial report / discussion: https://mail.python.org/pipermail/async-sig/2017-June/000293.html

[dimaqq]

@yuri, this bug doesn't require gather, here's a version with futures and explicit await's instead. It produces same output:

import asyncio


async def generator():
    while True:
        x = yield 42
        print("received", x)
        await asyncio.sleep(0.1)


async def user(name, g):
    print("sending", name)
    await g.asend(name)


async def helper():
    g = generator()
    await g.asend(None)

    u0 = asyncio.ensure_future(user("user-0", g))
    u1 = asyncio.ensure_future(user("user-1", g))
    u2 = asyncio.ensure_future(user("user-2", g))

    await u0
    await u1
    await u2


if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(helper())

Same with asyncio.get_event_loop().create_task as well.

[cjerdonek]

Note that the example can be further simplified by replacing user() with:

    async def send_hello(g):
        print("sending: hello")
        await g.asend("hello")

Then the output is:

sending: hello
received hello
sending: hello
sending: hello
received None
received None

[jancespivo]

I've reproduced the problem also in 3.7 branch.

import asyncio

loop = asyncio.get_event_loop()


async def consumer():
    while True:
        await asyncio.sleep(0)
        message = yield
        print('received', message)


async def amain():
    agenerator = consumer()
    await agenerator.asend(None)

    fa = asyncio.create_task(agenerator.asend('A'))
    fb = asyncio.create_task(agenerator.asend('B'))
    await fa
    await fb


loop.run_until_complete(amain())

Output:

received A
received None

If the line await asyncio.sleep(0) is omitted the output is ok:

received A
received B

[1st1]

Thanks Jan. Thanks a lot for a short script to reproduce this bug.

The actual problem here is that asynchronous generators don't control their 'asend' and 'athrow' coroutines in any way. So if you have two of them iterating *in parallel* they will cause their asynchronous generator to be in an inconsistent state.

The most obvious solution to this problem is to prohibit iterating 'asend'/'athrow' objects in parallel by throwing an exception.

Nathaniel, what are your thoughts on this?

[njsmith]

My thoughts: https://bugs.python.org/issue32526#msg309783

[1st1]

Thanks, I'll look into adding ag_running properly.

[1st1]

New changeset fc4a044 by Yury Selivanov in branch 'master':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (bpo-7468)
fc4a044

[1st1]

New changeset 2f87a7d by Yury Selivanov (Miss Islington (bot)) in branch '3.8':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (GH-7468) (bpo-16486)
2f87a7d