BUG: cythonization / compliation failure with development branch of cython

[tacaswell]

Describe your issue.

cython commit cython/cython@77918c5 in cython/cython#4670 broke scipy installation. This is currently only on cython's master branch and not in an alpha release yet.

I'm not sure if this is best fixed on the scipy or cython side.

I reported this via back-channels to @rgommers.

Reproducing Code Example

pip install -v --no-build-isolation .

with the development version of cython installed.

### Error message

```shell
The relevant error from compilation is :


  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      if name == NULL:
          name_copy = name
      else:
          name_copy = strdup(name)

      capsule = PyCapsule_New(func, name_copy, &raw_capsule_destructor)
                                               ^
  ------------------------------------------------------------

  /home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pyx:80:45: Cannot assign type 'void (*)(object) except *' to 'PyCapsule_Destructor'
  Traceback (most recent call last):
    File "/home/tcaswell/source/p/scipy/scipy/scipy/_build_utils/cythoner.py", line 28, in <module>
      main()
    File "/home/tcaswell/source/p/scipy/scipy/scipy/_build_utils/cythoner.py", line 20, in main
      sbp.run(['cython', '-3', '--fast-fail',
    File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cython', '-3', '--fast-fail', '--output-file', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-jnf3i747/build/scipy/_lib/_ccallback_c.cpython-312-x86_64-linux-gnu.so.p/_ccallback_c.c', '--include-dir', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-jnf3i747/build', '/home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pyx']' returned non-zero exit status 1.

### SciPy/NumPy/Python version information

numpy main, cpython main, scipy main

[da-woods]

You can probably fix this issue in Scipy by changing the definition of raw_capsule_destructor to

cdef void raw_capsule_destructor(object capsule) noexcept:

(This should be compatible with the latest 0.29.x release too)

In this case though I think Cython should update its PyCapsule definitions though. I'll consider that some more and do it if necessary. So perhaps wait for the Cython to update if it isn't urgent scipy works with the development branch.

We expect a small amount of breakage from this change though. But your specific case probably shouldn't be one.

[da-woods]

Right - having investigated a bit more:

The capsule destructor function is called in tp_dealloc of the capsule with the assumption that it doesn't raise a Python exception (there's no error checking afterwards)
https://github.com/python/cpython/blob/02389658a4751a0166e2ed22be112b646378a01b/Objects/capsule.c#L257

tp_dealloc in Python isn't supposed to raise an exception and if it does this is a fatal error:
https://github.com/python/cpython/blob/b9634ac776c24bc4d4a57859d884a94cdfe16043/Objects/object.c#L2386

Therefore the function raw_capsule_destructor really shouldn't raise an exception. In Cython 0.29.x that's the default for a cdef function that doesn't return a Python object or have an except specification. In Cython 3.0 cdef functions do raise exceptions by default. So the correct fix is to annotate it with noexcept.

Therefore I think this is expected.

[tacaswell]

With this patch:

$ git diff
diff --git a/scipy/_lib/_ccallback_c.pyx b/scipy/_lib/_ccallback_c.pyx
index 0704acc62..93a7ec73c 100644
--- a/scipy/_lib/_ccallback_c.pyx
+++ b/scipy/_lib/_ccallback_c.pyx
@@ -14,7 +14,7 @@ from .ccallback cimport (ccallback_t, ccallback_prepare, ccallback_release, CCAL
 # PyCapsule helpers
 #

-cdef void raw_capsule_destructor(object capsule):
+cdef void raw_capsule_destructor(object capsule) noexcept:
     cdef char *name
     name = PyCapsule_GetName(capsule)
     free(name)

Cythonization fails for 3.0.0a11 that the function definition is invalid cython and with cython master it fails with things that look like

 [201/1566] Generating 'scipy/special/_ellip_harm_2.cpython-312-x86_64-linux-gnu.so.p/_ellip_harm_2.c'.
  FAILED: scipy/special/_ellip_harm_2.cpython-312-x86_64-linux-gnu.so.p/_ellip_harm_2.c
  /home/tcaswell/.virtualenvs/bleeding/bin/python3 /home/tcaswell/source/p/scipy/scipy/scipy/_build_utils/cythoner.py ../../scipy/special/_ellip_harm_2.pyx scipy/special/_ellip_harm_2.cpython-312-x86_64-linux-gnu.so.p/_ellip_harm_2.c

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  from .special cimport cython_special
  ^
  ------------------------------------------------------------

  /home/tcaswell/source/p/scipy/scipy/scipy/special.pxd:1:0: 'scipy/special/special.pxd' not found
  Traceback (most recent call last):
    File "/home/tcaswell/source/p/scipy/scipy/scipy/_build_utils/cythoner.py", line 28, in <module>
      main()
    File "/home/tcaswell/source/p/scipy/scipy/scipy/_build_utils/cythoner.py", line 20, in main
      sbp.run(['cython', '-3', '--fast-fail',
    File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cython', '-3', '--fast-fail', '--output-file', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-psklxsqo/build/scipy/special/_ellip_harm_2.cpython-312-x86_64-linux-gnu.so.p/_ellip_harm_2.c', '--include-dir', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-psklxsqo/build', '/home/tcaswell/source/p/scipy/scipy/scipy/special/_ellip_harm_2.pyx']' returned non-zero exit status 1.

[da-woods]

Cythonization fails for 3.0.0a11

Yeah - it doesn't look like noexcept has made it into a alpha release yet (I thought it had, but misremembered).

/home/tcaswell/source/p/scipy/scipy/scipy/special.pxd:1:0: 'scipy/special/special.pxd' not found

This looks like a different issue. I suspect it comes from cython/cython@0c8dea1 but I'll need to look into it further.

[matusvalo]

/home/tcaswell/source/p/scipy/scipy/scipy/special.pxd:1:0: 'scipy/special/special.pxd' not found

The issue is that the special.pxd and special package have the same name.

This looks like a different issue. I suspect it comes from cython/cython@0c8dea1 but I'll need to look into it further.

Yes I confirm that this commit is source of issue.

[matusvalo]

I had another thought about the issue. If I am not wrong, the current behaviour is correct. E.g. in python following structure:

/mypg/
   __init__.py
   a/
       __init__.py
       b.py
   a.py

you are not able to import module a.py (At least I was not able to do that). Every import ends up loading a/__init__.py file.
In this repo, the fix is easy. Just migrate content from file special.pxd to file special/__init__.pxd - special/__init__.pxd should contain:

from . cimport cython_special

Note: I understand that this is breaking change and I am not sure what is the best way for users of Cython.

@da-woods @scoder what do you think?

[da-woods]

I don't have a lot of time to think about this this week unfortunately.

I think @matusvalo is broadly right that ideally there wouldn't be a file with the same name as a package folder. I believe these declarations are for users of Scipy rather than just Scipy itself so if we're proposing the fix is in Scipy then it probably does need to be backwards compatible and I haven't had a chance to look into that.

We could probably do a workaround in Cython if necessary (prefer special.pxd over special/__init__.pxd in some circumstances, to be decided...) but that'd definitely need more thought.

[rgommers]

Cython 3.0 beta 1 landed - and we haven't solved this issue yet, hence our pre-release CI job is failing (as reported in gh-18062). So I guess it's time to patch things up.

@tirthasheshpatel I have a memory of browsing a conversation last week where you were digging into this. But I can't find it - can you point it out, or am I imagining things?

[tirthasheshpatel]

@rgommers I think you are remembering #17360. The issue there was that a lot of files were being opened and it exceeded ulimit -n which is why some tests failed. But I think the issue here is quite different. @da-woods and @matusvalo's suggestions sound like they should address the issue here; adding noexpect to the definition of raw_capsule_destructor and moving special.pxd, optimize.pxd, and linalg.pxd to their __init__.pxd files.

I tried it locally but now I can other failures:

[279/1619] Generating 'scipy/special/cython_special.cpython-311-x86_64-linux-gnu.so.p/cython_special.c'.
FAILED: scipy/special/cython_special.cpython-311-x86_64-linux-gnu.so.p/cython_special.c 
/home/tirthasheshpatel/oss/virtualenvs/scipy-dev/bin/cython -3 --fast-fail --output-file scipy/special/cython_special.cpython-311-x86_64-linux-gnu.so.p/cython_special.c --include-dir . scipy/special/cython_special.pyx
/home/tirthasheshpatel/oss/virtualenvs/scipy-dev/lib/python3.11/site-packages/pythran/tables.py:4555: FutureWarning: In the future `np.bytes` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
  obj = getattr(themodule, elem)
warning: scipy/special/_complexstuff.pxd:22:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310

Error compiling Cython file:
------------------------------------------------------------
...
        # avoid under/overflows in intermediate results
        return exp(-lbeta(1 + n - k, 1 + k) - log(n + 1))
    elif k > 1e8*fabs(n):
        # avoid loss of precision
        num = Gamma(1 + n) / fabs(k) + Gamma(1 + n) * n / (2*k**2) # + ...
        num /= pi * fabs(k)**n
        ^
------------------------------------------------------------

scipy/special/orthogonal_eval.pxd:109:8: Calling gil-requiring function not allowed without gil
[284/1619] Generating 'scipy/special/_ufuncs.cpython-311-x86_64-linux-gnu.so.p/_ufuncs.c'.
FAILED: scipy/special/_ufuncs.cpython-311-x86_64-linux-gnu.so.p/_ufuncs.c 
/home/tirthasheshpatel/oss/virtualenvs/scipy-dev/bin/cython -3 --fast-fail --output-file scipy/special/_ufuncs.cpython-311-x86_64-linux-gnu.so.p/_ufuncs.c --include-dir . scipy/special/_ufuncs.pyx
/home/tirthasheshpatel/oss/virtualenvs/scipy-dev/lib/python3.11/site-packages/pythran/tables.py:4555: FutureWarning: In the future `np.bytes` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
  obj = getattr(themodule, elem)
warning: scipy/special/_complexstuff.pxd:22:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
warning: scipy/special/_lambertw.pxd:29:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
warning: scipy/special/_lambertw.pxd:30:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310

Error compiling Cython file:
------------------------------------------------------------
...
        # avoid under/overflows in intermediate results
        return exp(-lbeta(1 + n - k, 1 + k) - log(n + 1))
    elif k > 1e8*fabs(n):
        # avoid loss of precision
        num = Gamma(1 + n) / fabs(k) + Gamma(1 + n) * n / (2*k**2) # + ...
        num /= pi * fabs(k)**n
        ^
------------------------------------------------------------

scipy/special/orthogonal_eval.pxd:109:8: Calling gil-requiring function not allowed without gil

[andyfaff]

Would this be the remedy?

with gil:
   num /= pi * fabs(k)**n

[da-woods]

This looks to be to do with the new treatment of ** as maybe returning complex. The easiest thing would probably to decorate the function with @cython.cpow(True) to keep the old behaviour. I suspect you may well want the old behaviour anyway

I think the error is probably a mistake but it needs a bit of investigation.

[WarrenWeckesser]

This operation does not require the GIL. We can avoid the Cython issue (whether or not it is a Cython bug) by replacing the line with

num /= pi * pow(k, n)

and include pow in the functions cimported from libc.math.

There are two changes in that suggestion: replace a**b with pow(a, b), and drop the call of fabs around k. This line is only reached if k > 1e8*fabs(n), so we know k is positive. fabs(k) could be replaced with just k in the previous line, too. In fact, there is code below that checks if k > 0: that is within the code block for if k > 1e8*fabs(n):, so I wonder if there is a bug lurking here related to checking the sign of k.

Edit: In fact, I suspect there is a 10 year old mistake in this function. This relevant commit is 728b802, and the commit message is "BUG: special: deal with loss of precision in binom(n, k) for |k| >> |n| ", so I think the line elif k > 1e8*fabs(n): should be elif fabs(k) > 1e8*fabs(n):, but I haven't actually checked it carefully. If that is true, then we need fabs(k) in my suggested change. So the line that Cython 3.0.0b1 is complaining about can be changed to

num /= pi * pow(fabs(k), n)

[tirthasheshpatel]

I did that and got a lot of Cannot assign type 'function_definition except * nogil' to 'c_function_definition'. Looks like we will need to add a lot of noexcepts everywhere. Also, NumPy has a lot of exported Cython functions that might also need noexcept e.g. bitgen_t.next_double:

Error compiling Cython file:
------------------------------------------------------------
...

        if not PyCapsule_IsValid(capsule, capsule_name):
            raise ValueError("Invalid pointer to anon_func_state.")

        numpy_urng = <bitgen_t *> PyCapsule_GetPointer(capsule, capsule_name)
        unuran_urng = unur_urng_new(numpy_urng.next_double,
                                              ^
------------------------------------------------------------

/home/tirthasheshpatel/oss/scipy/scipy/stats/_unuran/unuran_wrapper.pyx:128:46: Cannot assign type 'double (*)(void *) except * nogil' to '_unur_urng_new_sampleunif_ft'

[ilayn]

If this is going to be only valid for double, I'd bite the bullet and do an if negative .... else .... branching. libc, cython, and gil stuff is not fun to play with. It becomes very annoying very quickly to have the simplest functions matching the dtype of scalars. It becomes much worse with complex.h

[matusvalo]

Looks like we will need to add a lot of noexcepts everywhere.

Just a note: Cython 3 has legacy_implicit_noexcept compiler directive which forces exception handling to be backward compatible with 0.29.X Cython. This directive was introduced to help Cython users with migration to Cython 3.

[rgommers]

In this repo, the fix is easy. Just migrate content from file special.pxd to file special/__init__.pxd - special/__init__.pxd should contain:

from . cimport cython_special

I had a look at this one - that fix doesn't actually seem to work in any form. It triggers one of these errors:

Error compiling Cython file:
------------------------------------------------------------
...
from . cimport cython_special
^
------------------------------------------------------------
scipy/special/__init__.pxd:1:0: 'scipy/cython_special.pxd' not found

or

from . cimport linalg, optimize, special
^
------------------------------------------------------------
scipy/__init__.pxd:1:0: relative cimport beyond main package is not allowed

I've tried leaving out a scipy/__init__.pxd, leaving it empty, or cimporting linalg & co. Plus absolute cimport instead of from . cimport xxx (which seems completely broken in 0.29.32/33). Note that a from . cimport in scipy/special/__init__.pxd looks for the specified pxd file in scipy rather than scipy/special. Absolute imports work, but I don't want to use those, too easy to pick up an installed version of the package instead.

Here is a branch if anyone feels like digging in more: https://github.com/scipy/scipy/compare/main...rgommers:scipy:cy30b1-fixes?expand=1.

Using relative cimport's with __init__.pxd files seems completely untested (see https://github.com/cython/cython/blob/master/tests/run/relative_cimport.srctree), unless I am missing it. @da-woods are you aware of a problem here, or am I missing something obvious?

[matusvalo]

Looking in your example you are having this:

cython<3.0b1 # unpin when gh-17234 is addressed

Which version of cython are you using? If cython 0.29.X the example won't work.

[scoder]

Absolute imports work, but I don't want to use those, too easy to pick up an installed version of the package instead.

Is it really? Normally, once scipy/__init__.pxd has been found and imported, it seems very unlikely that a subsequent import of a scipy.something would go to a completely different package directory. Why would it?

[tacaswell]

With the current master branch of cython

$ git describe
3.0.0b2-14-gdacb26cec

I am still seeing failures. Applying

diff --git a/scipy/_lib/_ccallback_c.pyx b/scipy/_lib/_ccallback_c.pyx
index 5208a15c2..42ef8ab55 100644
--- a/scipy/_lib/_ccallback_c.pyx
+++ b/scipy/_lib/_ccallback_c.pyx
@@ -6,7 +6,7 @@ from cpython.long cimport PyLong_AsVoidPtr
 from libc.stdlib cimport free
 from libc.string cimport strdup

-from .ccallback cimport (ccallback_t, ccallback_prepare, ccallback_release, CCALLBACK_DEFAULTS,
+from scipy._lib.ccallback cimport (ccallback_t, ccallback_prepare, ccallback_release, CCALLBACK_DEFAULTS,
                          ccallback_signature_t)


diff --git a/scipy/linalg.pxd b/scipy/linalg.pxd
index 1f656b870..c7c49f9ad 100644
--- a/scipy/linalg.pxd
+++ b/scipy/linalg.pxd
@@ -1 +1 @@
-from .linalg cimport cython_blas, cython_lapack
+from scipy.linalg cimport cython_blas, cython_lapack
diff --git a/scipy/optimize.pxd b/scipy/optimize.pxd
index 2402eeb02..51342b990 100644
--- a/scipy/optimize.pxd
+++ b/scipy/optimize.pxd
@@ -1 +1 @@
-from .optimize cimport cython_optimize
+from scipy.optimize cimport cython_optimize
diff --git a/scipy/optimize/cython_optimize.pxd b/scipy/optimize/cython_optimize.pxd
index d5a0bdd75..d35f8da68 100644
--- a/scipy/optimize/cython_optimize.pxd
+++ b/scipy/optimize/cython_optimize.pxd
@@ -7,5 +7,5 @@
 # support. Changing it causes an ABI forward-compatibility break
 # (gh-11793), so we currently leave it as is (no further cimport
 # statements should be used in this file).
-from .cython_optimize._zeros cimport (
+from scipy.optimize.cython_optimize._zeros cimport (
     brentq, brenth, ridder, bisect, zeros_full_output)
diff --git a/scipy/special.pxd b/scipy/special.pxd
index 62cb82807..1daa9fb37 100644
--- a/scipy/special.pxd
+++ b/scipy/special.pxd
@@ -1 +1 @@
-from .special cimport cython_special
+from scipy.special cimport cython_special
diff --git a/scipy/special/_legacy.pxd b/scipy/special/_legacy.pxd
index a97f56dcd..7697f9041 100644
--- a/scipy/special/_legacy.pxd
+++ b/scipy/special/_legacy.pxd
@@ -11,7 +11,7 @@ from libc.math cimport isnan, isinf, NAN

 from . cimport sf_error
 from ._ellip_harm cimport ellip_harmonic
-from .sph_harm cimport sph_harmonic
+from scipy.special.sph_harm cimport sph_harmonic
 from ._cephes cimport (bdtrc, bdtr, bdtri, expn, nbdtrc,
                        nbdtr, nbdtri, pdtri, kn, yn,
                        smirnov, smirnovi, smirnovc, smirnovci, smirnovp)
diff --git a/scipy/stats/_unuran/unuran_wrapper.pyx b/scipy/stats/_unuran/unuran_wrapper.pyx
index 5d4d8ebaa..0ddafadc2 100644
--- a/scipy/stats/_unuran/unuran_wrapper.pyx
+++ b/scipy/stats/_unuran/unuran_wrapper.pyx
@@ -8,7 +8,7 @@ from numpy.random cimport bitgen_t

 from scipy._lib.ccallback cimport ccallback_t
 from scipy._lib.messagestream cimport MessageStream
-from .unuran cimport *
+from scipy.stats._unuran.unuran cimport *
 import warnings
 import threading
 import functools

you
I'm seeing errors that look like:

  [287/1624] Generating 'scipy/special/cython_special.cpython-312-x86_64-linux-gnu.so.p/cython_special.c'.
  FAILED: scipy/special/cython_special.cpython-312-x86_64-linux-gnu.so.p/cython_special.c
  /home/tcaswell/.virtualenvs/bleeding/bin/cython -3 --fast-fail --output-file scipy/special/cython_special.cpython-312-x86_64-linux-gnu.so.p/cython_special.c --include-dir . scipy/special/cython_special.pyx
  WARNING  Overriding pythran description with argspec information for: numpy.random.binomial
...
  warning: scipy/special/_complexstuff.pxd:22:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
  warning: scipy/special/_loggamma.pxd:30:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
  warning: scipy/special/_loggamma.pxd:31:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
...


  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  from .orthogonal_eval cimport eval_chebyc as _func_eval_chebyc
  ctypedef double complex _proto_eval_chebyc_double_complex__t(double, double complex) nogil
  cdef _proto_eval_chebyc_double_complex__t *_proto_eval_chebyc_double_complex__t_var = &_func_eval_chebyc[double_complex]
  from .orthogonal_eval cimport eval_chebyc as _func_eval_chebyc
  ctypedef double _proto_eval_chebyc_double__t(double, double) nogil
  cdef _proto_eval_chebyc_double__t *_proto_eval_chebyc_double__t_var = &_func_eval_chebyc[double]
                                                                        ^
  ------------------------------------------------------------

  scipy/special/cython_special.pyx:1204:70: Cannot assign type 'double (*)(double, double) except * nogil' to '_proto_eval_chebyc_double__t *'

  [285/1624] Generating 'scipy/special/_ufuncs.cpython-312-x86_64-linux-gnu.so.p/_ufuncs.c'.
  FAILED: scipy/special/_ufuncs.cpython-312-x86_64-linux-gnu.so.p/_ufuncs.c
  /home/tcaswell/.virtualenvs/bleeding/bin/cython -3 --fast-fail --output-file scipy/special/_ufuncs.cpython-312-x86_64-linux-gnu.so.p/_ufuncs.c --include-dir . scipy/special/_ufuncs.pyx
  WARNING  Overriding pythran description with argspec information for: numpy.random.binomial
  WARNING  Overriding pythran description with argspec information for: numpy.random.bytes
...
  warning: scipy/special/_complexstuff.pxd:22:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
  warning: scipy/special/_lambertw.pxd:29:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
...


  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  from .orthogonal_eval cimport eval_chebyc as _func_eval_chebyc
  ctypedef double complex _proto_eval_chebyc_double_complex__t(double, double complex) nogil
  cdef _proto_eval_chebyc_double_complex__t *_proto_eval_chebyc_double_complex__t_var = &_func_eval_chebyc[double_complex]
  from .orthogonal_eval cimport eval_chebyc as _func_eval_chebyc
  ctypedef double _proto_eval_chebyc_double__t(double, double) nogil
  cdef _proto_eval_chebyc_double__t *_proto_eval_chebyc_double__t_var = &_func_eval_chebyc[double]
                                                                        ^
  ------------------------------------------------------------

  scipy/special/_ufuncs.pyx:1514:70: Cannot assign type 'double (*)(double, double) except * nogil' to '_proto_eval_chebyc_double__t *'

  [269/1624] Generating 'scipy/_lib/_ccallback_c.cpython-312-x86_64-linux-gnu.so.p/_ccallback_c.c'.
  FAILED: scipy/_lib/_ccallback_c.cpython-312-x86_64-linux-gnu.so.p/_ccallback_c.c
  /home/tcaswell/.virtualenvs/bleeding/bin/cython -3 --fast-fail --output-file scipy/_lib/_ccallback_c.cpython-312-x86_64-linux-gnu.so.p/_ccallback_c.c --include-dir . ../../scipy/_lib/_ccallback_c.pyx
  WARNING  Overriding pythran description with argspec information for: numpy.random.binomial
...
  warning: /home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pxd:5:83: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.
...
  warning: /home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pyx:109:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
  warning: /home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pyx:135:87: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.
...

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      if name == NULL:
          name_copy = name
      else:
          name_copy = strdup(name)

      capsule = PyCapsule_New(func, name_copy, &raw_capsule_destructor)
                                               ^
  ------------------------------------------------------------

  /home/tcaswell/source/p/scipy/scipy/scipy/_lib/_ccallback_c.pyx:80:45: Cannot assign type 'void (*)(object) except *' to 'PyCapsule_Destructor'

[matusvalo]

scipy/special/cython_special.pyx:1204:70: Cannot assign type 'double (*)(double, double) except * nogil' to '_proto_eval_chebyc_double__t *'

These errors are caused by new noexcept keyword. Cython 3 requires this annotation. The fix is easy - just add to function definition noexcept keyword. This is also 100% portable back to cython >= 0.29.31.

See http://docs.cython.org/en/latest/src/userguide/migrating_to_cy30.html#exception-values-and-noexcept

I am playing with building scipy with cython3 so I can provide PR for that.

[matusvalo]

I have created PoC pull request showing compilable version of scipy using cython3 - #18242

[rgommers]

Absolute imports work, but I don't want to use those, too easy to pick up an installed version of the package instead.

Is it really? Normally, once scipy/__init__.pxd has been found and imported, it seems very unlikely that a subsequent import of a scipy.something would go to a completely different package directory. Why would it?

So it seems both are broken - relative imports are clearly safer in principle, but in practice they're used less because historically they've been less robust.

It may seem unlikely, but it's currently broken in both master and 0.29.x when you use the new depfile support and you have an existing version of the package you are building installed. I've now ran into this with both SciPy and pandas. I'll get around to debugging that and opening an issue soon.