Adapt to C-API changes in CPython 3.13 #5767

scoder · 2023-10-15T18:07:56Z

No description provided.

…API).

da-woods · 2023-10-15T20:33:51Z

Looks OK except for running it on Github actions

…e from CPython directly.

scoder · 2023-10-16T09:06:19Z

except for running it on Github actions

Yeah, CPython 3.13a1 is only freshly released. It'll become available eventually. This isn't meant for 3.0.4.

hroncok · 2023-10-20T11:17:38Z

For the record, we tried to use this with Python 3.13.0a1 in Fedora and we see:

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:902:55: error: implicit declaration of function ‘_PyDict_GetItem_KnownHash’ [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:2707:22: error: implicit declaration of function ‘_PyList_Extend’; did you mean ‘PyList_Append’? [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:890:65: error: implicit declaration of function ‘_PyDict_NewPresized’ [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:815:39: error: implicit declaration of function ‘_PyThreadState_UncheckedGet’; did you mean ‘PyThreadState_GetUnchecked’? [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:3000:38: error: implicit declaration of function ‘_PyDict_SetItem_KnownHash’ [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:87139:12: error: implicit declaration of function ‘__Pyx_PyObject_GetOptionalAttr’; did you mean ‘PyObject_GetOptionalAttr’? [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:87784:28: error: implicit declaration of function ‘_PyVectorcall_Function’; did you mean ‘PyVectorcall_Function’? [-Werror=implicit-function-declaration]

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:92216:16: error: implicit declaration of function ‘_PyLong_FromByteArray’ [-Werror=implicit-function-declaration]

scoder · 2023-10-20T13:47:39Z

Apparently, there was a huge CPython core initiative earlier this summer to break existing Python packages using their existing "private by convention" APIs. I'm yet to understand why this happened, why so much effort from their side was put into deliberate breakage of existing code.

Some of the API functions are probably easy to replace, but we'll have to see how it goes. I don't see a replacement for _PyLong_FromByteArray() yet. I'll ask.

…ks a lot like "0x03000000", whereas "0x030d0000" is easier to distinguish.

…CPython 3.13. See python/cpython#106320

scoder · 2023-10-20T14:35:57Z

I think I got rid of the removed C-API calls. That was easy because all of them were using private C-API functions for which we have compatibility replacements that I just had to enable for Py3.13+. The replacement for _PyLong_FromByteArray() in particular is awful, though, and not everything is necessarily as fast as before. But it should compile now.

Cython/Utility/ModuleSetupCode.c

…s is gone in Py3.13).

hroncok · 2023-10-22T14:13:18Z

Unfortunately, I still have:

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c: In function ‘__Pyx_PyInt_As_long’:
  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Parsing.c:92478:23: error: implicit declaration of function ‘_PyLong_AsByteArray’ [-Werror=implicit-function-declaration]
  92478 |                 ret = _PyLong_AsByteArray((PyLongObject *)v,
        |                       ^~~~~~~~~~~~~~~~~~~

The rest is gone.

hroncok · 2023-10-22T14:16:01Z

I see there is a pull request open by you to get it back. I can wait, just wanted to report back.

Cython/Utility/ObjectHandling.c

hroncok · 2023-10-22T18:34:36Z

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/Code.c:105843:13: error: implicit declaration of function ‘_PyGen_SetStopIterationValue’; did you mean ‘__Pyx_PyGen__FetchStopIterationValue’? [-Werror=implicit-function-declaration]
  105843 |             _PyGen_SetStopIterationValue(result);
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
         |             __Pyx_PyGen__FetchStopIterationValue

hroncok · 2023-10-24T12:31:11Z

  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/FlowControl.c: In function ‘__Pyx_set_iter_next’:
  /builddir/build/BUILD/cython-3.0.4/Cython/Compiler/FlowControl.c:55703:19: error: implicit declaration of function ‘_PySet_NextEntry’ [-Werror=implicit-function-declaration]
  55703 |         int ret = _PySet_NextEntry(iter_obj, ppos, value, &hash);
        |                   ^~~~~~~~~~~~~~~~

Also fix a test output check in Py3.13+: the error message starts with a capital letter in CPython.

hroncok · 2023-10-27T11:12:49Z

I still get SystemError: Cannot modify a string currently used when trying to use this to build other things (e.g. lxml).

scoder · 2023-10-27T11:27:05Z

I still get SystemError: Cannot modify a string currently used when trying to use this to build other things (e.g. lxml).

Ok, I'll try it myself. Or do you have a quick indication where this happens? A stacktrace would help.

hroncok · 2023-10-27T11:32:57Z

I have this traceback:

Traceback (most recent call last):
  ...
  File "/builddir/build/BUILD/lxml-4.9.3/setupinfo.py", line 180, in ext_modules
    result = cythonize(result, compiler_directives=cythonize_directives)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/usr/lib64/python3.13/site-packages/Cython/Build/Dependencies.py", line 1300, in cythonize_one
    result = compile_single(pyx_file, options, full_module_name=full_module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Main.py", line 615, in compile_single
    return run_pipeline(source, options, full_module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Main.py", line 539, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 398, in run_pipeline
    data = run(phase, data)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 375, in run
    return phase(data)
           ^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 34, in parse
    tree = context.parse(source_desc, scope, pxd = 0, full_module_name = full_module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/Cython/Compiler/Main.py", line 381, in parse
    tree = Parsing.p_module(s, pxd, full_module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Cython/Compiler/Parsing.py", line 3912, in Cython.Compiler.Parsing.p_module
  File "Cython/Compiler/Parsing.py", line 3931, in Cython.Compiler.Parsing.p_module
  File "Cython/Compiler/Parsing.py", line 2460, in Cython.Compiler.Parsing.p_statement_list
  File "Cython/Compiler/Parsing.py", line 2409, in Cython.Compiler.Parsing.p_statement
  File "Cython/Compiler/Parsing.py", line 3611, in Cython.Compiler.Parsing.p_def_statement
  File "Cython/Compiler/Parsing.py", line 2485, in Cython.Compiler.Parsing.p_suite_with_docstring
  File "Cython/Compiler/Parsing.py", line 2460, in Cython.Compiler.Parsing.p_statement_list
  File "Cython/Compiler/Parsing.py", line 2429, in Cython.Compiler.Parsing.p_statement
  File "Cython/Compiler/Parsing.py", line 1908, in Cython.Compiler.Parsing.p_if_statement
  File "Cython/Compiler/Parsing.py", line 1922, in Cython.Compiler.Parsing.p_else_clause
  File "Cython/Compiler/Parsing.py", line 2474, in Cython.Compiler.Parsing.p_suite
  File "Cython/Compiler/Parsing.py", line 2481, in Cython.Compiler.Parsing.p_suite_with_docstring
  File "Cython/Compiler/Scanning.py", line 447, in Cython.Compiler.Scanning.PyrexScanner.next
  File "Cython/Plex/Scanners.py", line 126, in Cython.Plex.Scanners.Scanner.read
  File "Cython/Plex/Scanners.py", line 155, in Cython.Plex.Scanners.Scanner.scan_a_token
  File "Cython/Plex/Scanners.py", line 222, in Cython.Plex.Scanners.Scanner.run_machine_inlined
SystemError: Cannot modify a string currently used

scoder · 2023-10-27T11:59:39Z

It looks like the CPython devs also tightened the C-API checks, and while doing that, might have made a couple of … interesting decisions. I'll investigate, but I'm really annoyed by this.

hroncok · 2023-10-27T13:45:11Z

I've tried to use the pure Python version of Cython to build lxml instead.

This is what I get now:

ImportError: /builddir/build/BUILD/lxml-4.9.3/src/lxml/etree.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyBytes_Join

da-woods · 2023-10-27T17:55:50Z

SystemError: Cannot modify a string currently used

The missing check in unicode_modifiable (https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L1622) that isn't in our __Pyx_unicode_modifiable is

if (_PyUnicode_HASH(unicode) != -1)
        return 0;

Git blame says that this was added in 2011 so it isn't a new check, ~~but I guess they're generating hashes more eagerly.~~

The associated comment says

// copied directly from unicode_object.c "unicode_modifiable
// removing _PyUnicode_HASH since it's a macro we don't have
//  - this is OK because trying PyUnicode_Resize on a non-modifyable
//  object will still work, it just won't happen in place

I believe that's still true. I think what failing is

cython/Cython/Utility/ObjectHandling.c

Lines 3082 to 3086 in 526e1fc

    
           __Pyx_INCREF(*p_left); 
        
           // copy 'right' into the newly allocated area of 'left' 
        
           #if PY_VERSION_HEX >= 0x030D0000 
        
           if (unlikely(PyUnicode_CopyCharacters(*p_left, left_len, right, 0, right_len) < 0)) return NULL;

The failure is:

PyUnicode_CopyCharacters checks modifiable
The "incref" will give it a refcount of 2 which will make it unmodifiable.

So that seems like a reasonable failure. I think we just need to move the incref after. ~~Although actually I'm beginning to doubt the incref and I wonder if it should be a gotref~~ No - it's right - it accounts for the reference we return. I think it should be last thing

da-woods · 2023-10-27T18:02:12Z

Cython/Utility/ObjectHandling.c

@@ -3066,7 +3075,7 @@ static CYTHON_INLINE PyObject *__Pyx_PyUnicode_ConcatInPlaceImpl(PyObject **p_le
        __Pyx_INCREF(*p_left);

        // copy 'right' into the newly allocated area of 'left'
-        #if PY_VERSION_HEX >= 0x030D0000
+        #if PY_VERSION_HEX >= 0x030d0000
        if (unlikely(PyUnicode_CopyCharacters(*p_left, left_len, right, 0, right_len) < 0)) return NULL;
        #else
        _PyUnicode_FastCopyCharacters(*p_left, left_len, right, 0, right_len);


Can't propose a change because it goes outside what github considers the edited region, but this block should be:

} // copy 'right' into the newly allocated area of 'left' #if PY_VERSION_HEX >= 0x030d0000 if (unlikely(PyUnicode_CopyCharacters(*p_left, left_len, right, 0, right_len) < 0)) return NULL; #else _PyUnicode_FastCopyCharacters(*p_left, left_len, right, 0, right_len); #endif __Pyx_INCREF(*p_left); // Incref accounts for the returned reference to *p_left return *p_left;

The incref shouldn't happen on the return NULL error path I believe.

…ET_ITEM() in a valid use case. See python/cpython#106168

…reference counting of the result and to make sure we only resize and copy into strings to which we own the only reference. Background is that PyUnicode_CopyCharacters() gained an additional check in Py3.13a1 that conflicts with our previous safety net of early incref-ing the resized string. See cython#5767 (comment)

…s gone.

hroncok · 2023-10-29T17:02:01Z

I was able to build pyyaml (tests passed) and python-lxml (tests errored due to unrelated Python 3.13 incompatibility) with the pure Python version of Cython from this PR.

hroncok · 2023-10-29T17:02:19Z

I was also able to build Cython itself.

hroncok · 2023-10-29T17:25:54Z

And with the self-compiled Cython, I've successfully built a couple of other packages.

hroncok · 2023-10-29T17:55:19Z

ImportError: /builddir/build/BUILD/lxml-4.9.3/src/lxml/objectify.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyStack_AsDict

…y fail to build.

scoder · 2023-10-30T09:15:52Z

I think this gets close enough to looking good that we should include it in 3.0.5. The test suite builds with some test failures. Given that Py3.13 is only in it's earliest alpha stage, I think it's important to unblock dependent projects by having a Cython release that allows them to build and test.

I'll merge this and fix the remaining issues later.

hroncok · 2023-10-30T09:54:13Z

Thank you so much for getting this into a buildable state despite all the hurdles.

scoder added 5 commits October 15, 2023 14:53

CI: Start testing against Python 3.13.

1d4fde3

CI: Start using new C-API functions in CPython 3.13 (also in Limited …

dd7019b

…API).

CI: Test Limited C-API functions in CPython 3.13.

7021b07

Fix C function signature.

45dfbbb

CI: Allow failures in CPython 3.13.

6bb6c79

scoder added enhancement Code Generation labels Oct 15, 2023

scoder added this to the 3.0.x milestone Oct 15, 2023

scoder added 2 commits October 16, 2023 10:58

Fix variable usage and add comment for error case.

733737d

Avoid defining our own redundant helper function if we can use the on…

f47ae0a

…e from CPython directly.

nitzmahone mentioned this pull request Oct 17, 2023

libyaml-based builds segfault on Python 3.13.0a1 yaml/pyyaml#761

Open

scoder added 4 commits October 20, 2023 16:12

Fix wrong function name usage.

03eff8c

Fix typo.

57245a8

Make PY_HEX_VERSION check for Py3.13 easier to read: "0x030D0000" loo…

77ab583

…ks a lot like "0x03000000", whereas "0x030d0000" is easier to distinguish.

Remove usages of private CPython C-API functions that disappeared in …

7001531

…CPython 3.13. See python/cpython#106320

da-woods reviewed Oct 20, 2023

View reviewed changes

Cython/Utility/ModuleSetupCode.c Show resolved Hide resolved

CI: Install latest pip and setuptools in Py3.11+ (also since distutil…

22f785a

…s is gone in Py3.13).

scoder changed the title ~~Adapt to new C-API features in CPython 3.13~~ Adapt to C-API changes in CPython 3.13 Oct 22, 2023

da-woods reviewed Oct 22, 2023

View reviewed changes

Cython/Utility/ObjectHandling.c Show resolved Hide resolved

Disable _PyLong_AsByteArray() usage in Py3.13. It was removed in 3.13a1.

1917732

Avoid calling _PyGen_SetStopIterationValue() in Py3.13+.

0c4f686

scoder added 2 commits October 27, 2023 12:41

Avoid using _PyObject_NextNotImplemented is Py3.13.

ee31668

Fix version guard: Py3.13, not Py3.11.

5816828

Also fix a test output check in Py3.13+: the error message starts with a capital letter in CPython.

Fix typo in test output.

0de8aaf

da-woods reviewed Oct 27, 2023

View reviewed changes

scoder added 3 commits October 29, 2023 12:15

Work around a change in CPython 3.13a1 that prevents calling PyList_S…

8b9c854

…ET_ITEM() in a valid use case. See python/cpython#106168

Make __Pyx_PyBytes_Join() work in Python 3.13 where _PyBytes_Join() i…

047f264

…s gone.

Disable usage of _PyLong_Copy() in Py3.13.

ebc9d28

scoder added 3 commits October 29, 2023 21:05

Disable usage of _PyStack_AsDict() in Py3.13.

fe9117d

Do not install any test dependencies in Python 3.13 for now since the…

b538f76

…y fail to build.

Remove unused utility code dependency.

99a6797

scoder modified the milestones: 3.0.x, 3.0.5 Oct 29, 2023

scoder merged commit fd96443 into cython:master Oct 30, 2023
83 checks passed

scoder deleted the py313_capi branch October 30, 2023 09:16

hroncok mentioned this pull request Oct 31, 2023

BUG: error: implicit declaration of function ‘_Py_HashDouble’ (building with Python 3.13~alpha1) numpy/numpy#25035

Closed

isc-adang mentioned this pull request Nov 1, 2023

[autoupdate] Update Cython from 3.0.4 to 3.0.5 intersystems/iknow#371

Merged

hroncok mentioned this pull request Nov 1, 2023

[BUG] Python 3.13 C++17: error: standard attributes in middle of decl-specifiers (CYTHON_UNUSED) #5790

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt to C-API changes in CPython 3.13 #5767

Adapt to C-API changes in CPython 3.13 #5767

scoder commented Oct 15, 2023

da-woods commented Oct 15, 2023

scoder commented Oct 16, 2023

hroncok commented Oct 20, 2023

scoder commented Oct 20, 2023

scoder commented Oct 20, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 24, 2023

hroncok commented Oct 27, 2023

scoder commented Oct 27, 2023

hroncok commented Oct 27, 2023

scoder commented Oct 27, 2023

hroncok commented Oct 27, 2023

da-woods commented Oct 27, 2023 •

edited

da-woods Oct 27, 2023 •

edited

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

scoder commented Oct 30, 2023

hroncok commented Oct 30, 2023

Adapt to C-API changes in CPython 3.13 #5767

Adapt to C-API changes in CPython 3.13 #5767

Conversation

scoder commented Oct 15, 2023

da-woods commented Oct 15, 2023

scoder commented Oct 16, 2023

hroncok commented Oct 20, 2023

scoder commented Oct 20, 2023

scoder commented Oct 20, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 22, 2023

hroncok commented Oct 24, 2023

hroncok commented Oct 27, 2023

scoder commented Oct 27, 2023

hroncok commented Oct 27, 2023

scoder commented Oct 27, 2023

hroncok commented Oct 27, 2023

da-woods commented Oct 27, 2023 • edited

da-woods Oct 27, 2023 • edited

Choose a reason for hiding this comment

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

hroncok commented Oct 29, 2023

scoder commented Oct 30, 2023

hroncok commented Oct 30, 2023

da-woods commented Oct 27, 2023 •

edited

da-woods Oct 27, 2023 •

edited