-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change contextvars C API to use PyObject #78943
Comments
Unfortunately, the current C API for PEP-567 has a flaw: it uses non-PyObject pointers. This causes problems with integrating with tools like Cython, where PyObject is special and a lot of machinery recognizes it and manages refcounts correctly. It also goes against the recent push to improve C API; one of the things we want to fix is to eliminate non-PyObject pointer types from public APIs entirely. Because this C API is new (landed in 3.7.0) I think it might make sense to change it in 3.7.1. I *think* this is a relatively safe (albeit annoying) change:
If we don't change this API now, we'll likely have to either live with it, or face a very slow deprecation period. |
Just to add to this issue: I originally realized that something is wrong with the design when we had a super hard to track memory leak in uvloop, caused by Cython being unable to automatically manage increfs/decrefs for PyContext* pointers. So I do believe this is a serious pitfall. Adding Guido to the nosy list as he accepted the PEP and IMO still has a say in this. |
IMHO it's not too late to change the public C API in Python 3.7.1, since it's a very new API, I don't expect many users, and I only expect compiler warnings nor hard error if existing code pass PyContextVar* instead of PyObject*. I agree that it's way better to use PyObject* rather than specific PyContextVar*. The C API should not leak implementation details ;-) |
Let’s change it ASAP. It’s still up to Ned whether to hold up 3.7.1, if he On Fri, Sep 21, 2018 at 8:28 AM STINNER Victor <report@bugs.python.org>
-- |
I concur that the sooner the change is applied, the better it will be for everyone. We'll need to make a prominent notice in the release notes though. |
Something can be added at: |
Yeah, thank you for suggestion, I'll do that. I'll also update the PEP, docs in other places, and try to spread the word. Now I'm just waiting for Ned to confirm if this goes into 3.7.1 or 3.7.2. It's his say. |
A notable exception is Py_buffer. [1] As well, cross-interpreter data must not be PyObject, though that isn't much of an issue (yet). :) At some point it may make sense to make _PyCrossInterpreterData [2] part of the public C-API. However, we can cross *that* bridge later. :) Using PyObject for contextvars makes sense (for the reasons you described) as long as they won't be shared between interpreters. I'm not super familiar with the contextvars C-API, but it looks like cross-interpreter issues are *not* a factor. If that's the case then there's nothing further to discuss. :) [1] https://docs.python.org/3/c-api/buffer.html#buffer-structure |
Right, because Py_buffer isn't a PyObject at all :)
Yeah, PyContext, PyContextVar, and PyContextToken aren't supposed to be shared between sub-interpreters directly. Context is essentially a mapping of Context Variables to arbitrary Python Objects, so sharing it transparently isn't possible. |
It owns a PyObject reference to the buffer owner, though. |
Since we've already delayed 3.7.1rc cutoff for a few days, let's get it into 3.7.1 if you have the time, Yury. |
PEP update: python/peps@977a94d |
Fixed in master and 3.7. Thanks! |
Perhaps this change caused new compiler warnings: In file included from ./Include/pytime.h:6:0,
from ./Include/Python.h:68,
from /home/serhiy/py/cpython/Modules/_asynciomodule.c:1:
/home/serhiy/py/cpython/Modules/_asynciomodule.c: In function ‘_asyncio_Task___init___impl’:
./Include/object.h:895:14: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
(op) = (op2); \
^
/home/serhiy/py/cpython/Modules/_asynciomodule.c:1954:5: note: in expansion of macro ‘Py_XSETREF’
Py_XSETREF(self->task_context, PyContext_CopyCurrent());
^~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c: In function ‘init_current_context’:
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:1503:44: warning: passing argument 1 of ‘PyContextVar_Set’ from incompatible pointer type [-Wincompatible-pointer-types]
PyContextToken *tok = PyContextVar_Set(current_context_var, tl_context);
^~~~~~~~~~~~~~~~~~~
In file included from ./Include/Python.h:116:0,
from /home/serhiy/py/cpython/Modules/_decimal/_decimal.c:29:
./Include/context.h:63:24: note: expected ‘PyObject * {aka struct _object *}’ but argument is of type ‘PyContextVar * {aka struct _pycontextvarobject *}’
PyAPI_FUNC(PyObject *) PyContextVar_Set(PyObject *var, PyObject *value);
^~~~~~~~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:1503:27: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
PyContextToken *tok = PyContextVar_Set(current_context_var, tl_context);
^~~~~~~~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c: In function ‘current_context’:
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:1517:26: warning: passing argument 1 of ‘PyContextVar_Get’ from incompatible pointer type [-Wincompatible-pointer-types]
if (PyContextVar_Get(current_context_var, NULL, &tl_context) < 0) {
^~~~~~~~~~~~~~~~~~~
In file included from ./Include/Python.h:116:0,
from /home/serhiy/py/cpython/Modules/_decimal/_decimal.c:29:
./Include/context.h:56:17: note: expected ‘PyObject * {aka struct _object *}’ but argument is of type ‘PyContextVar * {aka struct _pycontextvarobject *}’
PyAPI_FUNC(int) PyContextVar_Get(
^~~~~~~~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c: In function ‘PyDec_SetCurrentContext’:
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:1564:44: warning: passing argument 1 of ‘PyContextVar_Set’ from incompatible pointer type [-Wincompatible-pointer-types]
PyContextToken *tok = PyContextVar_Set(current_context_var, v);
^~~~~~~~~~~~~~~~~~~
In file included from ./Include/Python.h:116:0,
from /home/serhiy/py/cpython/Modules/_decimal/_decimal.c:29:
./Include/context.h:63:24: note: expected ‘PyObject * {aka struct _object *}’ but argument is of type ‘PyContextVar * {aka struct _pycontextvarobject *}’
PyAPI_FUNC(PyObject *) PyContextVar_Set(PyObject *var, PyObject *value);
^~~~~~~~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:1564:27: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
PyContextToken *tok = PyContextVar_Set(current_context_var, v);
^~~~~~~~~~~~~~~~
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c: In function ‘PyInit__decimal’:
/home/serhiy/py/cpython/Modules/_decimal/_decimal.c:5542:25: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
current_context_var = PyContextVar_New("decimal_context", NULL);
^ |
Right, I'll make a PR. |
Thank you Serhiy, for re-opening this! I've pushed fixes to 3.7 and master branches. |
The PyContext struct is not intended to be public, and users of the API don't need anything more specific than PyObject. Also see pythongh-78943.
The PyContext struct is not intended to be public, and users of the API don't need anything more specific than PyObject. Also see pythongh-78943.
The PyContext struct is not intended to be public, and users of the API don't need anything more specific than PyObject. Also see pythongh-78943.
The PyContext struct is not intended to be public, and users of the API don't need anything more specific than PyObject. Also see pythongh-78943.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: