deadlock with asyncio+contextmanager+ExitStack

[oconnor663]

BPO	25779
Nosy	@gvanrossum, @ncoghlan, @vstinner, @1st1, @oconnor663
Superseder	bpo-25782: CPython hangs on error context set to the error itself

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/1st1'
closed_at = <Date 2015-12-02.17:03:22.875>
created_at = <Date 2015-12-02.15:37:25.706>
labels = ['type-bug', 'invalid', 'expert-asyncio']
title = 'deadlock with asyncio+contextmanager+ExitStack'
updated_at = <Date 2015-12-02.20:42:08.228>
user = 'https://github.com/oconnor663'

bugs.python.org fields:

activity = <Date 2015-12-02.20:42:08.228>
actor = 'yselivanov'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2015-12-02.17:03:22.875>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2015-12-02.15:37:25.706>
creator = 'oconnor663'
dependencies = []
files = []
hgrepos = []
issue_num = 25779
keywords = ['3.5regression']
message_count = 13.0
messages = ['255719', '255721', '255722', '255726', '255728', '255730', '255732', '255733', '255748', '255749', '255765', '255777', '255778']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ncoghlan', 'vstinner', 'yselivanov', 'oconnor663']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = '25782'
type = 'behavior'
url = 'https://bugs.python.org/issue25779'
versions = ['Python 3.5']

[oconnor663]

The following hangs at 100% CPU on Python 3.5, though not on Python 3.4:

1. Start an asyncio coroutine with run_until_complete().
2. Inside the coroutine, enter an ExitStack using a with-statement.
3. Inside the with-statement, call ExitStack.enter_context() with a generator context manager. It doesn't matter what the generator yields.
4. After the enter_context() call, raise an exception.

Here's an example script that does all of this and repros the hang: https://gist.github.com/oconnor663/483db2820bb5f877c9ed

[gvanrossum]

Interestingly, it doesn't hang when you raise a different error. There's some new code dealing with the RuntimeError coming out of a generator if it raises StopIteration (instead of returning) introduced by issue bpo-22906. Yury, it looks like you introduced that? (The diff is this:

changeset: 95932:36a8d935c322
user: Yury Selivanov <yselivanov@sprymix.com>
date: Sat May 09 11:44:30 2015 -0400
summary: PEP-479: Change StopIteration handling inside generators.

[1st1]

Trying to reproduce without contextstack.

[1st1]

Here's a minimal test to reproduce:

    import reprlib


    def main():
        if 0:
            yield
        raise RuntimeError


    m = main()
    try:
        m.send(None)
    except RuntimeError as ex:
        ex.__context__ = ex
        reprlib.repr(ex)
        

Looks like it's a bug in reprlib. It's not related to PEP 492/479.

It's also reproducible in Python 3.4 and 3.3.

Nick, ExitStack does this (indirectly) 'ex.__context__ = ex' thing -- I think that's a bug of contextlib.

[1st1]

Created another issue for the reprlib bug: bpo-25781. It appears we don't even need a generator:

    import reprlib

try:
    raise RuntimeError
except RuntimeError as ex:
    ex.__context__ = ex
    reprlib.repr(ex)

Closing this one with "not a bug".

[1st1]

It's not even a reprlib bug:

try:
raise Exception
except Exception as ex:
ex.__context__ = ex
hasattr(1, 'aa')

[oconnor663]

Thanks for chasing this down. Yury, can you suggest a workaround?

[1st1]

Thanks for chasing this down. Yury, can you suggest a workaround?

I'm not sure how to workaround this :( Hopefully we can fix this in 3.5.1.

[1st1]

FWIW the bug was identified in bpo-25782. I've drafted a patch to fix it, please review.

[1st1]

The question is whether we should raise an exception or not:

   ex.__context__ = ex

[1st1]

Another issue for contextlib: http://bugs.python.org/issue25786

[oconnor663]

Yury, can you help me understand why hasattr("foo", "bar") triggers the infinite loop there, but not print("foo")?

[1st1]

Yury, can you help me understand why hasattr("foo", "bar") triggers the infinite loop there, but not print("foo")?

hasattr uses getattr under the hood. getattr raises an AttributeError, and that triggers PyErr_SetError, which has an infinite "while" loop. Instead of "hasattr" you can use anything that raises an error.