Use PyErr_FormatUnraisable() on Python 3.13 #34

vstinner · 2023-11-15T17:28:43Z

Use the new public PyErr_FormatUnraisable() on Python 3.13.

The private _PyErr_WriteUnraisableMsg() function was removed in Python 3.13:
python/cpython#111643

vstinner · 2023-11-15T17:31:30Z

src/c/_cffi_backend.c

@@ -6109,6 +6108,18 @@ static void _my_PyErr_WriteUnraisable(PyObject *t, PyObject *v, PyObject *tb,
    if (extra_error_line == NULL)
        extra_error_line = "";

+#if PY_VERSION_HEX >= 0x030D0000
+    if (obj != NULL) {
+        PyErr_FormatUnraisable("Exception ignored %c%s%R%s",


Maybe it's an opportunity to enhance the error message?

Should it be "Exception ignored ...", "Exception ignored in ..." (add in ) or "Exception ignored: ..." (add : )?

vstinner · 2023-11-15T17:32:11Z

cc @serhiy-storchaka @arigo

vstinner · 2023-11-15T17:34:35Z

By the way, _PyThreadState_UncheckedGet() should be replaced with the new public PyThreadState_GetUnchecked() function.

serhiy-storchaka

You forget to add PyErr_Restore().

arigo · 2023-11-15T19:26:16Z

src/c/_cffi_backend.c

+                               first_char, objdescr + 1, extra_error_line);
+    }
+#else
+    PyObject *s;


This does not compile on older Visual Studio versions, but I'm unsure how old these need to be. You can't declare a variable anywhere else than at the start of a block in older versions of C.

Ok, fixed. I made the assumption that cffi is built by a C99 compiler these days.

I'm trying to keep the code compiling for 2.7, and that might require much older VS on Windows.

I'm trying to keep the code compiling for 2.7

@arigo I'm hoping to convince you that that's not a realistic goal in 2023 with the resources available to this project 😉 . It's clearly still possible to make it work, but keeping test/CI infrastructure alive that's capable of actually verifying that (before breaking changes get merged) is already prohibitively expensive, and getting more so by the day. For the past few years, it's not even possible to download the 2.7 Python build tooling from Microsoft anymore- I keep a private stash of some of those tools around because I still have to deal with it occasionally, but that's not really workable for modern GHA runners. My general approach is "if it's not being tested, it should be assumed broken", and I can't justify spending any time trying to test or fix that. ☹️

I'm trying hard for provide a best effort support for Python 2.7 in my https://pythoncapi-compat.readthedocs.io/ project. I had C code to support VS2015 which doesn't support C99 static inline. But I removed it, it made the code harder to maintain. Moreover, GitHub Action doesn't provide Python 2.7. I'm now considering to officially remove Python 2 support and remove Python 2 code in pythoncapi-compat.

Yep, I've tried "best effort" support like this so many times, but even with the best of intentions, it's effectively always broken when not backed by automated testing. It also ends up actively discouraging contributions (like this one!)...

@nitzmahone Sure, I see the point, but in this case the final result is that the pull request was changed to 4 obvious lines instead of a more complex #if dance that missed originally a PyErr_Restore(), in addition to breaking a compiler (that may or may not be too old to consider). I think it's a win for a project like CFFI.

vstinner · 2023-11-15T19:27:12Z

You forget to add PyErr_Restore().

Oops, fixed.

arigo · 2023-11-15T19:35:50Z

@serhiy-storchaka has a point. And I'd be happier with some change that avoids making two versions of the formatting code. As far as I'm concerned, killing a semi-private API is a minor annoyance that I'd be more happy to work around, instead of trying to rewrite more CFFI logic, because that ends up making duplication. So I wouldn't kill PyUnicode_FromFormat() just because we appear to be able to in some CPython version.

Use the new public PyErr_FormatUnraisable() on Python 3.13. The private _PyErr_WriteUnraisableMsg() function was removed in Python 3.13: python/cpython#111643

vstinner · 2023-11-15T20:01:13Z

@arigo:

@serhiy-storchaka has a point. And I'd be happier with some change that avoids making two versions of the formatting code. As far as I'm concerned, killing a semi-private API is a minor annoyance that I'd be more happy to work around, instead of trying to rewrite more CFFI logic, because that ends up making duplication. So I wouldn't kill PyUnicode_FromFormat() just because we appear to be able to in some CPython version.

Sure, it makes sense. I rewrote my PR to also use the s string on Python 3.13. The patch is now just 4 lines.

@arigo: Would you mind to review the updated PR?

arigo · 2023-11-16T08:17:17Z

Sure. Now I have no hesitation to merge the pull request. @nitzmahone, can you please confirm if I should "Enable auto-merge (squash)" for this kind of case?

nitzmahone · 2023-11-16T16:02:51Z

can you please confirm if I should "Enable auto-merge (squash)" for this kind of case?

Yeah, that's fine- basically "I'm happy with the code, merge on my behalf if all checks pass" vs babysitting it...

vstinner · 2023-11-16T16:30:21Z

Thanks for the review and merging my change! I first wanted to enhance the error message, but I understand the desire to have the same message on all Python versions and it makes sense.

Ah yeah, I missed PyErr_Restore(), since I'm used to call PyErr_WriteUnraisable() while an exception in set, not on a function which gets an exception as an argument. Hopefully, it was catched by review and fixed ;-)

nitzmahone · 2023-11-20T19:51:20Z

@vstinner @arigo Bleh- I threw together a manylinux_2_28 container with 3.13.0a1 in it and hacked up cibuildwheel to work with it (in hopes of wiring up a 3.13 prerelease canary build here), and it's failing under 3.13 with an unresolved symbol for PyErr_FormatUnraisable...

~~Shouldn't this have been PyErr_WriteUnraisable, not PyErr_FormatUnraisable?~~ (EDIT: NM this question, was looking at 3.12 docs). Seems like the underlying problem is that the new function is not exported by libpython in 3.13.0a1...

I don't see any libpython exports for FormatUnraisable:

(tmp-ad2272c74e493a7) [mdavis@mdavis-p1 lib]$ objdump -t libpython3.13.so | grep 'PyErr_.*Unrais'
00000000002237e0 l     F .text  00000000000001b8              _PyErr_WriteUnraisableDefaultHook
00000000002239a0 g     F .text  00000000000007aa              _PyErr_WriteUnraisableMsg
0000000000224150 g     F .text  000000000000000a              PyErr_WriteUnraisable

nitzmahone · 2023-11-20T19:55:30Z

(or maybe the new libpython export didn't make it into 3.13.0a1 yet)- in any case, this doesn't appear to be fully working yet...

nitzmahone · 2023-11-20T20:25:13Z

OK, looks like this will be fixed in 3.13.0a2 (I just built against the top of main, which I assume will be a2 tomorrow-ish):

(tmp-2fcf66c976036a9) [mdavis@mdavis-p1 tmp-2fcf66c976036a9]$ objdump -t ~/.pyenv/versions/3.13-dev/lib/libpython3.13.so | grep PyErr_.*Unrai
0000000000245d00 l     F .text  00000000000001b8              _PyErr_WriteUnraisableDefaultHook
0000000000245ec0 g     F .text  0000000000000096              PyErr_FormatUnraisable
0000000000245f60 g     F .text  0000000000000009              PyErr_WriteUnraisable

nitzmahone · 2023-11-20T20:52:38Z

... and yep, full test suite passes against that build of "almost-alpha2", so I think we're good to go here...
============ 1951 passed, 89 skipped, 4 xfailed, 10 warnings in 255.53s (0:04:15) ============

vstinner mentioned this pull request Nov 15, 2023

gh-108082: Remove _PyErr_WriteUnraisableMsg() python/cpython#111643

Merged

vstinner commented Nov 15, 2023

View reviewed changes

vstinner mentioned this pull request Nov 15, 2023

gh-112026: Add again _PyThreadState_UncheckedGet() function python/cpython#112121

Merged

serhiy-storchaka reviewed Nov 15, 2023

View reviewed changes

arigo reviewed Nov 15, 2023

View reviewed changes

Use PyErr_FormatUnraisable() on Python 3.13

b4a4ef1

Use the new public PyErr_FormatUnraisable() on Python 3.13. The private _PyErr_WriteUnraisableMsg() function was removed in Python 3.13: python/cpython#111643

vstinner force-pushed the unraisable branch from 1a3c5fe to b4a4ef1 Compare November 15, 2023 19:58

arigo enabled auto-merge (squash) November 16, 2023 08:15

arigo disabled auto-merge November 16, 2023 08:15

arigo merged commit 49127c6 into python-cffi:main Nov 16, 2023
53 checks passed

vstinner deleted the unraisable branch November 16, 2023 16:28

beagold mentioned this pull request Apr 14, 2024

Release request (1.17.0) #71

Open

tseaver mentioned this pull request Apr 18, 2024

refactor: convert 'setup()' metadata to TOML zopefoundation/persistent#195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use PyErr_FormatUnraisable() on Python 3.13 #34

Use PyErr_FormatUnraisable() on Python 3.13 #34

vstinner commented Nov 15, 2023

vstinner Nov 15, 2023

vstinner commented Nov 15, 2023

vstinner commented Nov 15, 2023

serhiy-storchaka left a comment

arigo Nov 15, 2023

vstinner Nov 15, 2023

arigo Nov 15, 2023

nitzmahone Nov 15, 2023

vstinner Nov 15, 2023

nitzmahone Nov 15, 2023

arigo Nov 16, 2023

vstinner commented Nov 15, 2023

arigo commented Nov 15, 2023

vstinner commented Nov 15, 2023

arigo commented Nov 16, 2023

nitzmahone commented Nov 16, 2023

vstinner commented Nov 16, 2023

nitzmahone commented Nov 20, 2023 •

edited

nitzmahone commented Nov 20, 2023

nitzmahone commented Nov 20, 2023

nitzmahone commented Nov 20, 2023

Use PyErr_FormatUnraisable() on Python 3.13 #34

Use PyErr_FormatUnraisable() on Python 3.13 #34

Conversation

vstinner commented Nov 15, 2023

Choose a reason for hiding this comment

vstinner commented Nov 15, 2023

vstinner commented Nov 15, 2023

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented Nov 15, 2023

arigo commented Nov 15, 2023

vstinner commented Nov 15, 2023

arigo commented Nov 16, 2023

nitzmahone commented Nov 16, 2023

vstinner commented Nov 16, 2023

nitzmahone commented Nov 20, 2023 • edited

nitzmahone commented Nov 20, 2023

nitzmahone commented Nov 20, 2023

nitzmahone commented Nov 20, 2023

nitzmahone commented Nov 20, 2023 •

edited