Adapt to exc_info changes in Py3.11

[scoder]

Py3.11 alpha has been changed to keep only a single exception value internally in sys.exc_info instead of a (type, value, traceback) tuple. There are a couple of places in Cython where we touch those fields and we should a) adapt and b) make sure we don't unnecessarily swap dead fields in Py3.11.

If this stays in CPython until the release, it's probably worth also backporting the necessary changes to 0.29.x.

First reported in #4499 (comment)

https://bugs.python.org/issue45711

[TeamSpen210]

There's some additional discussion here that indicates that (aside from two undocumented edge cases) the type and tb will always match the value, so Cython could actually just always discard the redundant pointers.

[scoder]

Yes, that's the idea.

[tacaswell]

The failures from this look like: (from installing cython itself)

    gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c -o build/temp.linux-x86_64-3.11/home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.o
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c: In function ‘__Pyx_PyErr_GetTopmostException’:
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23059:21: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23059 |     while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
          |                     ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23059:51: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23059 |     while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
          |                                                   ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c: In function ‘__Pyx__ExceptionSave’:
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23073:21: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23073 |     *type = exc_info->exc_type;
          |                     ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23075:19: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
    23075 |     *tb = exc_info->exc_traceback;
          |                   ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c: In function ‘__Pyx__ExceptionReset’:
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23089:24: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23089 |     tmp_type = exc_info->exc_type;
          |                        ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23091:22: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
    23091 |     tmp_tb = exc_info->exc_traceback;
          |                      ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23092:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23092 |     exc_info->exc_type = type;
          |             ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23094:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
    23094 |     exc_info->exc_traceback = tb;
          |             ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c: In function ‘__Pyx__GetException’:
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23151:28: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23151 |         tmp_type = exc_info->exc_type;
          |                            ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23153:26: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
    23153 |         tmp_tb = exc_info->exc_traceback;
          |                          ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23154:17: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
    23154 |         exc_info->exc_type = local_type;
          |                 ^~
    /home/tcaswell/source/other_source/cython/Cython/Compiler/Scanning.c:23156:17: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
    23156 |         exc_info->exc_traceback = local_tb;
          |                 ^~
    error: command '/lib/ccache/bin/gcc' failed with exit code 1

[YoSTEALTH]

Another victim to this bug: https://bugs.python.org/issue46428

Edit: was using Cython 3.0.0a10 btw

[da-woods]

With reference to the comment in the duplicate issue

I would rather have this fixed in the master branch first and wait with a release in 0.29.x until there is at least a beta release of CPython 3.11. I'd like to avoid churn in the legacy 0.29.x branch.

I suspect this may generate complaints - I think a good chunk of the performance tests needed for "have we made Python faster" depend on Cython 0.29.x, so there will be people chasing a compatible release

[hroncok]

The same concern applies to Fedora. We want to test our packages with Python 3.11 pre-releases, but can hardly do that without working Cython. We can backport patches from a draft pull request or whatnot, but backporting them directly from the master branch is usually quite tricky.

[scoder]

Let's get something clean into the master branch first. And then see what needs to be done to apply the same to 0.29.x, and what the impact is.

[scoder]

I wonder if it's enough for 0.29.x to disable CYTHON_FAST_THREAD_STATE in Py3.11 – in case that switch works correctly there. It would disable more than necessary, but should avoid the direct access to exc_info. As usual, users who want speed and features can switch to Cython 3.

Turns out, no, _PyErr_StackItem is also used in Cython's coroutines and isn't protected by the feature flag there (obviously).

[CaselIT]

As usual, users who want speed and features can switch to Cython 3.

Is v3 currently ok for production use, or since it's in alpha it's still in the "expect bugs"/"use at your own peril" phase? Maybe question is more like is it already used by popular libraries?

The context is that for sqlalchemy v2 we have replaced c extension with cython versions, and we wanted to start testing for py311 (since fedora is already reporting bugs for that version). Thanks!

cc @zzzeek

[mattip]

Is v3 currently ok for production use, or since it's in alpha it's still in the "expect bugs"/"use at your own peril" phase?

One problem with depending on cython v3 is that it cannot be used on conda-forge, so if you depend on it for a release, it will make building your package there more complicated: either you will have to package generated c files or add a "install from source" step to the recipe to get v3 into the environment.

[scoder]

Is v3 currently ok for production use, or since it's in alpha it's still in the "expect bugs"/"use at your own peril" phase?

Cython 3.0 is still in alpha because it is a) not feature complete yet and b) we reserve the right to fix bugs also in backwards incompatible ways before the final release, if that seems necessary, despite of our usual goal to keep things working for our users.

But as always with Cython, if a specific version works for you, it won't randomly stop working later. You can try it out yourself.

One problem with depending on cython v3 is that it cannot be used on conda-forge

Is there a reason why conda-forge cannot add it?

(Including the generated C files in your source distribution is always an option, and tends to be easy to do, although we are now moving away from recommending it in order to make it easier for packagers to move to a newer Cython version that supports newer CPython releases etc.)

[da-woods]

But as always with Cython, if a specific version works for you, it won't randomly stop working later. You can try it out yourself.

Although of course for Python 3.11 this isn't true, because changes to Python will probably break current Cython versions...

[scoder]

But as always with Cython, if a specific version works for you, it won't randomly stop working later.

... for the Python versions that it supports and was tested with. It's not like that has ever been different, with any version of Cython or Python. Both are live projects that change over time.

[scoder]

In the master branch, the exception handling seems to work in Py 3.11 now, with #4584 merged. There is still an issue with line tracing, which might be related to exception handling or not. And a couple of C compiler warnings that are worth looking into.

Regarding the backport 0.29.x, I tried to put the new implementations in #4584 next to the old ones to make the code clearer and easier to maintain, but it is quite a large change overall. I'd be happy to find something simpler (such as disabling feature flags), but would at least let this mature a bit before we consider backporting the whole change to 0.29.x.

[mattip]

One problem with depending on cython v3 is that it cannot be used on conda-forge

Is there a reason why conda-forge cannot add it?

Probably not, there is conda-forge/cython-feedstock#75 to try to get it going

[hrnciar]

I've tried to backport #4584 (see [0]) on top of 0.29.26 and managed to build Cython with it. But I ran into a segfault during the build of lxml 4.7.1 [1]. I've tried to debug it using gdb, but I cannot tell what's causing the failure. Maybe my backport is incorrect or there is some other problem in Cython. See [2] for full gdb output. Could you please take a look and help me to find out what's causing the failure?

[0] 0001-Backport-4584.txt
[1] https://download.copr.fedorainfracloud.org/results/@python/python3.11/fedora-rawhide-x86_64/03247525-python-lxml/builder-live.log.gz
[2] gdb-output.txt

[CaselIT]

@mattip @scoder thanks for the explanation. I guess we will wait for the next v3 alpha that includes the py3.11 change and give it a try. Thanks!

[zzzeek]

just because I know zero about anything "conda" , does this limitation mean that if SQLAlchemy depended on cython3, that SQLAlchemy could not be included in the anaconda distribution ? something like that?

[scoder]

I've tried to backport #4584 (see [0]) on top of 0.29.26 and managed to build Cython with it. But I ran into a segfault during the build of lxml 4.7.1 [1]. I've tried to debug it using gdb, but I cannot tell what's causing the failure. Maybe my backport is incorrect or there is some other problem in Cython. See [2] for full gdb output. Could you please take a look and help me to find out what's causing the failure?

#3  0x00007ffff704e795 in __Pyx_Coroutine_ResetFrameBackpointer (exc_state=0x7ffff6a8b6d0) at src/lxml/etree.c:265100
#4  __Pyx_Coroutine_SendEx (self=0x7ffff6a8b6b0, value=<optimized out>, closing=<optimized out>)

I'm sure there were changes in this area in the master branch. I remember doing some cleanups and fixing several issues with coroutines for recent CPython features such as limited API support, extension type specs and what not. Can't say how much of that went into 0.29.x. Probably not everything, given that 0.29.x is pretty much in bug-fix-only mode and has been for a while. I'm not saying that that's necessarily the issue here, but I'm not surprised that the change is not stand-alone.

[vstinner]

Cython 0.29.x is currently incompatible with the development version of Python 3.11, but this issue has been closed. Should a new issue be opened about Cython 0.29.x, or can this issue be reopened?

The Python change on exceptions breaks Cython which breaks numpy. I proposed to revert the change in Python until Cython is ready for the change: https://bugs.python.org/issue45711#msg412264

I also opened a discussion on python-dev: "Please update Cython before introcuding C API incompatible changes in Python".
https://mail.python.org/archives/list/python-dev@python.org/thread/RS2C53LDZPXHRR2VCY2G2YSPDVA4LNQU/

[dvarrazzo]

Hello! I wonder if you have a timeline to release Cython 3.0.0a11 including this fix (and #4721). As far as I could see, they are the only blockers to release psycopg 3 for Python 3.11 🙂

psycopg/psycopg#305

Thank you for your work!

[riemanli]

@vstinner if you build Cython 0.29.x with CFLAGS="-DCYTHON_FAST_THREAD_STATE=0 -DCYTHON_USE_EXC_INFO_STACK=0" python3.11 setup.py build_ext --inplace -f it successfully builds itself. I think there's some coroutine code that still needs fixing, but I think it's likely a significant chunk of Cython libraries will also work with those flags (it looks sufficient to build Numpy, for example).

I'd propose getting those settings into 0.29.x by default now (because it'll fix the majority of incompatibilities I think). Obviously that isn't fully compatible so it isn't "job done" but it's an easy start

Is this still the solution to this issue when using 0.29.x?

[scoder]

Is this still the solution to this issue when using 0.29.x?

When using 0.29.x and not 3.0.x, then yes, looks like that's needed. I'll see if I can get a 0.29.x point release out that does this automatically.

[scoder]

Actually, no, this is fixed also in 0.29.28 and later. Still, it's worth using Cython 3.x instead.

See #4610