Add Python 3.11 alpha 6 support #1872
Conversation
|
Draft PR: Python 3.11a6 has not been released yet. |
* On Python 3.11a6 and newer, get the PyFrameObject structure from
the internal C API ("internal/pycore_frame.h").
* On Python 3.9 and newer, use PyFrame_GetBack() and
PyFrame_GetCode().
* Add frame getter and setter functions to greenlet:
* get_f_code(frame)
* set_f_lineno(frame, lineno)
* set_f_code(frame, code)
* greenlet.h: the CFrame type has been renamed to _PyCFrame.
It has been released: https://pythoninsider.blogspot.com/2022/03/python-3110a6-is-available.html |
The PyFrame_GetCode() function is missing: |
|
That's referencing 3.6. So it needs to be ifdef'ed out for older Python versions? |
|
Please don't merge this yet, as the API changes are not final: https://discuss.python.org/t/proposal-rename-pyinterpreterframe-struct-as-py-framedata/14213/7 |
|
Don't worry about my previous comment - we'll make sure code that works against 3.11a6 keeps working until faster-cpython/ideas#309 provides a proper public API for the relevant frame manipulation tasks (although feel free to chime in there on the public API that you'd like to see to avoid having to delve into frame object internals) |
|
@ncoghlan Will wheels build for 3.11a6 continue to work? |
|
Yes, after looking at this PR and Victor's other PRs from the 3.11a6 changes, I realised that any change I proposed needed to be both source and binary compatible to avoid causing excessive hassle for the performance work (even if I might wish the situation were otherwise) |
|
Please merge. |
Sadly, some CI tests are failing :-( So far, I failed to understand why it's failing. It seems like a real bug in my PR, but I don't know how to reproduce it (see my previous comments). |
|
I've tried using this patch in Fedora together with both greenlet 2.0.0a2 and with greenlet 1.1.2 + backported Python 3.11 support. With greenlet 2.0.0a2, I was hitting python-greenlet/greenlet@5bbd0fc and worked around it. Either way, I ended up with |
Can you link to where that error is generated? (Presumably somewhere in greenlet?) That message is likely correct and may point to a solution (rebuild something using newer header files)? |
Not sure what you mean exactly. The output/traceback i get is: Both greenlet and gevent are built against the same headers from Python 3.11.0b3. But Cython was built on a previous Python release, let me try to rebuild that one on b3 as well if it makes a difference. |
No difference. |
I meant what source file raises that error. I haven't found it myself (I don't see it in the CPython source tree), but it occurs a lot in StackOverflow, usually for a numpy type. Since it isn't Cython either, what other tool or library could be raising this? |
|
|
I'm guessing something that's generated by Cython needs to be regenerated. Just using a newer Cython doesn't fix that. |
|
We already had: I've added: Still getting the same error. I am out of ideas. |
|
Me too. :-( Try reading the code that gets executed during that |
|
If I eliminate the gcd import, I get: It keeps saying file "src/gevent/greenlet.py", line 1 and the Cython-annotated file shows: I'm afraid this is too many layers deep for me for 1 AM. Thanks @gvanrossum, for the pointers anyway. I will sleep on it and will bother @vstinner when he is back from Kernel Recipes :) |
|
This is generated by Cython: __Pyx_ImportType(__pyx_t_2, "types", "CodeType", sizeof(PyCodeObject), __Pyx_ImportType_CheckSize_Warn);Hence, the code in https://github.com/cython/cython/blob/6ac2422b48b689b021a48dff9ee14095232baafe/Cython/Utility/ImportExport.c#L515 and further more or less does: result = PyObject_GetAttrString(PyImport_ImportModule("types"), "CodeType");
basicsize = ((PyTypeObject *)result)->tp_basicsize;
if ((size_t)basicsize < sizeof(PyCodeObject)) { ... error ... }
See https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#external-extension-types for an explanation for the check. |
|
In Python 3.11.0b3: PyObject *result;
result = PyObject_GetAttrString(PyImport_ImportModule("types"), "CodeType");
return PyLong_FromLong(((PyTypeObject *)result)->tp_basicsize);Returns 160. return PyLong_FromLong(sizeof(PyCodeObject));Returns 168. No extra headers, just pure CPython. So either |
|
Apparently PyCodeObject *result = PyCode_NewEmpty("", "", 0);
return (PyObject*)result;>>> import types
>>> type(reporoducer.gimme_pycodeobject()) is types.CodeType
True |
|
The issue is that struct PyCodeObject {
... (lots of fields) ...
char co_code_adaptive[1];
}and the size fields are then: .tp_basicsize = offsetof(PyCodeObject, co_code_adaptive),
.tp_itemsize = sizeof(_Py_CODEUNIT),(code simplified for clarity) This is new in 3.11, but The basicsize docs say:
The wording could be relaxed (or include Anyway, Cython checks that the struct it's using fits in I guess Cython should skip/relax the check if tp_itemsize is nonzero? (Edit: this check can be controlled using Cython's |
|
Thanks! If I do that (itemsize.patch for Cython 0.29.30), the next problem is: Which is indeed true, |
| cdef void* f_back | ||
| IF PY311A6: | ||
| cdef extern from "internal/pycore_frame.h": | ||
| ctypedef class types._PyInterpreterFrame [object _PyInterpreterFrame]: |
There was a problem hiding this comment.
I was thinking I could do something like:
| ctypedef class types._PyInterpreterFrame [object _PyInterpreterFrame]: | |
| ctypedef struct _PyInterpreterFrame: |
But that way, I get:
Error compiling Cython file:
------------------------------------------------------------
...
pass
IF PY311A6:
cdef extern from "internal/pycore_frame.h":
ctypedef struct _PyInterpreterFrame:
cdef CodeType f_code
^
------------------------------------------------------------
src/gevent/_gevent_cgreenlet.pxd:63:12: Expected an identifier, found 'cdef'
And when I remove cdef from cdef CodeType f_code as well, I get:
C struct/union member cannot be a Python object
There was a problem hiding this comment.
In if I cascade with using structs:
cdef extern from "Python.h":
ctypedef struct PyCodeObject:
pass
...
cdef extern from "internal/pycore_frame.h":
ctypedef struct _PyInterpreterFrame:
PyCodeObject* f_code
I end up with:
cdef inline void set_f_code(FrameType frame, CodeType code):
IF PY311A6:
frame.f_frame.f_code = code
^
------------------------------------------------------------
src/gevent/_gevent_cgreenlet.pxd:114:31: Cannot convert Python object to 'PyCodeObject *'
And I am getting into a seemingly neverending hole.
There was a problem hiding this comment.
This makes it compile:
diff --git a/src/gevent/_gevent_cgreenlet.pxd b/src/gevent/_gevent_cgreenlet.pxd
index 246773e..47669dc 100644
--- a/src/gevent/_gevent_cgreenlet.pxd
+++ b/src/gevent/_gevent_cgreenlet.pxd
@@ -57,13 +57,16 @@ cdef extern from "Python.h":
ctypedef class types.CodeType [object PyCodeObject]:
pass
+ ctypedef struct PyCodeObject:
+ pass
+
IF PY311A6:
cdef extern from "internal/pycore_frame.h":
- ctypedef class types._PyInterpreterFrame [object _PyInterpreterFrame]:
- cdef CodeType f_code
+ ctypedef struct _PyInterpreterFrame:
+ PyCodeObject* f_code
ctypedef class types.FrameType [object PyFrameObject]:
- cdef _PyInterpreterFrame f_frame
+ cdef _PyInterpreterFrame* f_frame
# Accessing the f_lineno directly doesn't work. There is an accessor
# function, PyFrame_GetLineNumber that is needed to turn the raw line number
# into the executing line number.
@@ -108,7 +111,7 @@ cdef inline CodeType get_f_code(FrameType frame):
cdef inline void set_f_code(FrameType frame, CodeType code):
IF PY311A6:
- frame.f_frame.f_code = code
+ frame.f_frame.f_code = <PyCodeObject *>code
ELSE:
frame.f_code = codeBut it fails on rutime with:
File "src/gevent/greenlet.py", line 689, in gevent._gevent_cgreenlet.Greenlet.spawn
g = cls(*args, **kwargs)
File "src/gevent/greenlet.py", line 301, in gevent._gevent_cgreenlet.Greenlet.__init__
self.spawning_stack = _extract_stack(self.spawning_stack_limit)
File "src/gevent/greenlet.py", line 164, in gevent._gevent_cgreenlet._extract_stack
set_f_lineno(older_Frame.f_lineno, get_f_lineno(frame)) # pylint:disable=undefined-variable
TypeError: Cannot convert int to frame
|
Maybe @brandtbucher can help here? |
Perhaps, but maybe somebody could clarify the exact problem that we're trying to solve at this point in the journey? From what I understand at this point, the issue is that Cython isn't recognizing
I'm worried I'm likely just repeating what people might already know, though. I've also never used Cython, so lines like |
|
I reported the issue to Cython: cython/cython#4827 I suggest to continue discussing the Cython there, since this is a PR on the gevent project ;-) |
|
Hm, the contents of CodeType aren't actually used, are they? |
|
Quoted from your Python issue:
One thing you can do is include literal c code: Of course the arguments still need to be types that Cython understands, but you may find that lets you hide some of the details from Cython |
|
Alright, @encukou reverted some of the changes here and replaced them with others. I've rebased and pushed the result to https://github.com/hroncok/gevent/commits/py311b3 and will test with that in Fedora. Note that there are 2 unresolved problems - tets failures/errors: We might leak some references: And greenlet names have changed: |
|
Thank you all for your help and patience. #1908, based on this PR and passing all tests on 3.11rc2, has been merged and released as gevent-22.8.0; binary wheels will be in the process of building and uploading for the next several hours. |
|
Wow, this is really cool! Thanks @jamadden for your nice project! It's great that it's ready before Python 3.11 final release. |
|
FYI we (Fedora) maintainted a downstream patch to unblock Python 3.11 until a new gevent version would be released: https://src.fedoraproject.org/rpms/python-gevent/blob/rawhide/f/master...hroncok:py311b3.patch |
do we still need it with 22.08.0 release or just 21.12 |
On Python 3.11a6 and newer, get the PyFrameObject structure from
the internal C API ("internal/pycore_frame.h").
On Python 3.9 and newer, use PyFrame_GetBack() and
PyFrame_GetCode().
Add frame getter and setter functions to greenlet:
greenlet.h: the CFrame type has been renamed to _PyCFrame.