BUG: Build failure with Cython 3.0.0b3 if scipy is already installed

[mgorny]

Describe your issue.

When using Cython 3 to build scipy in an environment where it is installed already, it fails to build as pasted below. Judging from the error message, it seems to be using cython_optimize.pxd from the installed tree rather than the source directory and looking for _zeros.pxd in path containing duplicate cython_optimize component.

The problem doesn't occur with Cython < 3, nor with Cython 3 if scipy is not yet installed.

This is causing problems when building the package for Gentoo. However, I was also able to reproduce it in pure venv, on top of git commit 9816141.

Reproducing Code Example

python3.10 -m venv .venv
. .venv/bin/activate
pip install --pre Cython
pip install build meson-python numpy==1.22.4 pybind11 pythran scipy wheel
python -m build -w -n --config-setting=setup-args=-Dblas=blas --config-setting=setup-args=-Dlapack=lapack

Error message

[1554/1587] Generating 'scipy/optimize/cython_optimize/_zeros.cpython-310-x86_64-linux-gnu.so.p/_zeros.c'.
FAILED: scipy/optimize/cython_optimize/_zeros.cpython-310-x86_64-linux-gnu.so.p/_zeros.c 
/tmp/scipy/.venv/bin/cython -3 --fast-fail --output-file scipy/optimize/cython_optimize/_zeros.cpython-310-x86_64-linux-gnu.so.p/_zeros.c --include-dir . scipy/optimize/cython_optimize/_zeros.pyx

Error compiling Cython file:
------------------------------------------------------------
...

# The following cimport statement provides legacy ABI
# support. Changing it causes an ABI forward-compatibility break
# (gh-11793), so we currently leave it as is (no further cimport
# statements should be used in this file).
from .cython_optimize._zeros cimport (
^
------------------------------------------------------------

/tmp/scipy/.venv/lib/python3.10/site-packages/scipy/optimize/cython_optimize.pxd:10:0: 'scipy/optimize/cython_optimize/cython_optimize/_zeros.pxd' not found
[1563/1587] Compiling C++ object scipy/sparse/sparsetools/_sparsetools.cpython-310-x86_64-linux-gnu.so.p/bsr.cxx.o
ninja: build stopped: subcommand failed.

SciPy/NumPy/Python version and system information

(note this is from installed scipy)

1.11.1 1.21.6 sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0)
/tmp/scipy/.venv/lib/python3.10/site-packages/scipy/__config__.py:146: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "0.29.35",
      "commands": "cython"
    },
    "c++": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.13.1",
      "include directory": "../../tmp/pip-build-env-hy2eo89f/overlay/lib/python3.10/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "openblas",
      "found": true,
      "version": "0.3.21.dev",
      "detection method": "pkgconfig",
      "include directory": "/usr/local/include",
      "lib directory": "/usr/local/lib",
      "openblas configuration": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2",
      "pc file directory": "/usr/local/lib/pkgconfig"
    },
    "lapack": {
      "name": "openblas",
      "found": true,
      "version": "0.3.21.dev",
      "detection method": "pkgconfig",
      "include directory": "/usr/local/include",
      "lib directory": "/usr/local/lib",
      "openblas configuration": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2",
      "pc file directory": "/usr/local/lib/pkgconfig"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.10.4",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/opt/python/cp310-cp310/bin/python",
    "version": "3.10"
  }
}

[rgommers]

Ugh yes, I need to follow up on that still. Relative imports in Cython are broken, and I'm fairly sure I figured out last time around that that is due to the recently-added depfile support (of course, there could be other reasons too). I'd expect this to be broken with Cython 0.29.35 as well, but not with 0.29.32 (depfile support was backported to .33 or .34).

Thanks for the report @mgorny.

[mgorny]

Thanks for confirming that. FYI, it seems to build fine for me with 0.29.35.

[h-vetinari]

Ugh yes, I need to follow up on that still. Relative imports in Cython are broken, and I'm fairly sure I figured out last time around that that is due to the recently-added depfile support

I'm a bit confused here. @mgorny reports that the error happens with Cython 3, where relative imports should be functional (modulo bugs, and this looks like one?).

[rgommers]

"functional modulo bugs" == broken (in this case)

[h-vetinari]

"functional modulo bugs" == broken (in this case)

You recently added a test case to Cython to capture some of our setup. Is there a cython issue for the breakage we're encountering here? I'm pretty sure the Cython devs would be receptive for solving this.

[rgommers]

I don't think that there is any test case; it's quite tricky to set up a test for this since it's a failure mode that only shows up when there's a matching import in site-packages and that import has a mismatch with the local version of what one is importing.

[h-vetinari]

@scoder @da-woods @matusvalo
Do you think it would make sense to open an issue on the cython repo for this? It might be hard to formulate in a test, but it's a regression (broadly speaking) compared to 0.29.

[scoder]

Could you explain why there is both a cython_optimize.pxd file and a cython_optimize package? Why isn't the first in cython_optimize/__init__.pxd?

[matusvalo]

Why isn't the first in cython_optimize/init.pxd?

I suppose this is workaround for an issue from Cython 0.29.X. Suppose following file structure:

########## b/__init__.py ##########

########## b/__init__.pxd ##########
from .bb cimport ULong

########## b/bb.pxd ##########
ctypedef unsigned long ULong

########## a.pyx ##########
from .b cimport ULong

cdef ULong x = 10
print(x)

In this file structure compiling a.pyx fails in Cython 0.29.X due issue cython/cython#3442. This was fixed in Cython 3.0.

I suppose (not 100% sure), scipy uses a workaround for using relative imports in packages is to use instead of __init__.pxd file with the same name as package. The fix of previous example in Cython 0.29.X is following:

########## b/__init__.py ##########

########## b/bb.pxd ##########
ctypedef unsigned long ULong

########## b.pxd ##########
from .b.bb cimport ULong

########## a.pyx ##########
from .b cimport ULong

cdef ULong x = 10
print(x)

Here compiling a.pyx is successful.

[matusvalo]

OK, I think I have fix. The issue seems to be in scipy build mechanism, not in Cython - see PR for details.

[rgommers]

The issue seems to be in scipy build mechanism, not in Cython

Isn't it still a bug that a relative cimport ends up picking up a .pxd file from site-packages? I think Cython should not be looking there to begin with.

The build passes on all CI jobs if scipy isn't already installed in site-packages. And if relative imports worked as they should, it would not make a difference at all whether there's a scipy packages in site-packages or not.

[matusvalo]

Isn't it still a bug that a relative cimport ends up picking up a .pxd file from site-packages?

Yes I agree this is at least strange behaviour. I need to further investigate it.

[rgommers]

Fixed for now in gh-18810, which should be backported to 1.11.2 soon. The follow-up should be on cython/cython#5511.