-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deprecate sys.set_coroutine_wrapper and replace it with more focused API(s) #76772
Comments
sys.set_coroutine_wrapper is a problematic API. It was added to solve a specific problem: asyncio debug mode wanting to track where coroutine objects are created, so that when unawaited coroutines are GC'ed, the warning they print can include a traceback. But to do this, it adds a *very* generic hook that lets you hook into the coroutine creation code to replace all instances of a built-in type with an arbitrary new type, which is highly unusual. It's hard to use correctly (see bpo-30578, and bpo-24342 before it). It's hard to use without creating performance problems, since anything you do here tends to add overhead to every async function call, and in many cases also every async function frame suspend/resume cycle. Fortunately, it's explicitly documented as provisional. (I think Yury intentionally excluded it from the stabilization of the rest of asyncio and async/await, because of the reasons above.) And the things we *actually* want to do here are very simple. The only known use cases are the one mentioned above (see asyncio.coroutines.CoroWrapper), and the one in bpo-30491. So after discussions with Yury, I'm proposing to migrate those both into the interpreter as directly usable, fast, built-in features that can be used individually or together, and deprecate sys.set_coroutine_wrapper. See bpo-30491 for details on that use case; here I'll discuss my plan for replacing the "coroutine origin tracking" that asyncio debug mode does. We add a new function sys.set_coroutine_origin_tracking(depth), and I guess sys.get_coroutine_origin_tracking() because why not. The depth is thread-local, and defaults to 0, which means that newly created coroutines don't capture any origin info. When a coroutine is created, it will check the current origin_tracking depth, and capture that many frames of traceback information. Like the current asyncio debug code and unlike real tracebacks, we won't capture actual frame objects -- we'll just capture (filename, lineno, function) information, to avoid pinning objects in memory. This is a debug helper, so it should avoid changing semantics whenever possible. Later, we can retrieve the captured information by accessing the new attribute coro.origin. In addition, the code in CoroutineType.__del__ that currently emits a warning for unawaited coroutines, will be modified to check for whether 'self' has origin information, and print it if so, similar to the current asyncio debug wrapper. Some particular points where I'd appreciate feedback: Should we add an envvar to configure coroutine source tracking? What about an -X switch? Should it be coro.origin or coro.cr_origin? On the one hand the cr_ prefixing is super annoying and I think we should probably get rid of it; on the other maybe we should keep it here so things stay consistent and then have a separate the debate about removing it in general. Most important: what format should we use for storing the origin information? I can see three plausible approaches:
Given this analysis I guess I'm leaning towards just calling StackSummary.extract, but I don't feel like I fully understand the consequences of calling into a Python module like that so want to hear what others think. |
+1 on on the API and for adding it in 3.7 and for deprecating set_coroutine_wrapepr. This will simplify asyncio code and other event loops (like uvloop) and will lead to less overhead on asyncio debug mode. I always disliked the set_coroutine_wrapper API. Even at the time when I added it to PEP-492 I understood that we'll have to replace it eventually with something more sensible. I think we should start working on a patch. Nathaniel, do you have time to champion the work?
I'd be strongly against calling I'm not sure how exactly the warning print logic will work with a list of tuples in cr_origin, I propose to start working on the implementation and figure that out during the review process. |
I think the simplest thing is probably to write the warning code in Python and stash it in Lib/_corohelper.py or something. Warnings already go through warnings.py (in particular, the warning printing code is still warnings.showwarning, it's never been rewritten in C), so we're already transitioning to Python at this stage anyway. |
I'd put the helper that accepts a list of traceback-like tuples and emits a warning in warnings.py and a C function to call it in warnings.c. |
Guido, I'd like to go forward with this and merge Nathaniel's PR. Quick summary:
Are you OK with this? |
Yes, I'm OK with this plan. Note that I've only read Nathaniel's original |
Merged! Thank you, Nathaniel! |
The following snippet crashes with SIGABRT: import asyncio
async def f(): pass
asyncio.gather(f()) llvm tells that the crash is in _PyGen_Finalize/_PyErr_WarnUnawaitedCoroutine |
Right, I *knew* I should be nervous about calling into Python from a C-level destructor... what's happening is:
We can get a similar crash by doing: import sys
async def f(): pass
sys.corocycle = [f]
sys.corocycle.append(sys.corocycle) If you run the same code on 3.6, then it gets collected at the same time, and it issues a warning using the regular PyErr_WarnEx. It turns out that code is even *more* careful and defensive and notices that the interpreter is being shutdown, so it just skips printing the warning entirely. I guess what we have to do is add a similar check to _PyErr_WarnUnawaitedCoroutine. You can imagine how excited I am that I started working on this patch so I could make sure people get more reliable notice of unawaited coroutines (bpo-30491), and not only has that been rejected, but now I'm writing code to explicitly hide unawaited coroutine warnings. Just saying'... |
Maybe it's better to roll this back and take a deep breath. The feature freeze should not be treated as an excuse to cram in things at the last minute. Before this week I had never heard of this proposal. And the sarcasm is a bad start of your stint as a core dev, Nathaniel. |
Nathaniel's latest PR looks good and fixes the crash. FWIW losing some warnings during interpreter shutdown isn't exactly a new problem, therefore I think there's nothing wrong with the new API here. For instance, I've seen asyncio's CoroWrapper losing warnings in the exact same fashion. So I've merged Nathaniel's PR. I think we can rollback the new API if we discover more problems during the beta-1 period. |
OK. I hope both of you keep an eye on this and actively try to test and |
That's the plan! |
Nathaniel, test_coroutines raises a bunch of Runtime and Deprecation warnings after this PR. Could you please make a PR to silence them? /Users/yury/dev/pydev/cpython/Lib/test/test_coroutines.py:1032: RuntimeWarning: coroutine 'CoroutineTest.test_await_12.<locals>.coro' was never awaited |
Specifically "DeprecationWarning: get_coroutine_wrapper is deprecated" needs to be silenced as part of this issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: