"No module named 'Cython'", but only when installing a package and only with pip 23.1 or later

[Zabolekar]

Bug report

Bug description:

I stumbled upon the issue while trying to build a package that requires Cython. I have seen it on Linux and Windows. The following minimal example has been created on Linux.

There are two files, a.pyx and setup.py:

# a.pyx
def f():
    pass

# setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize

extensions = [Extension("a", ["a.pyx"])]
setup(ext_modules=cythonize(extensions))

Creating the environment, activating it, installing Cython:

python3.12 -m venv 312
. 312/bin/activate
pip install Cython

Versions of everything installed:

$ python --version
Python 3.12.3
$ pip list
Package Version
------- -------
Cython  3.0.10
pip     24.0

Verifying that Cython can be imported:

$ python -c "import Cython; print(Cython.__version__)"
3.0.10

Trying to install the package itself:

$ pip install -e .
Obtaining file:///home/ssyekirin/Desktop/cython_issue
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/home/ssyekirin/Desktop/cython_issue/312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/ssyekirin/Desktop/cython_issue/312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/ssyekirin/Desktop/cython_issue/312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-1d0hsqit/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 448, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-1d0hsqit/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-1d0hsqit/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-1d0hsqit/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-1d0hsqit/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 2, in <module>
      ModuleNotFoundError: No module named 'Cython'
      [end of output]
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

As you can see, Cython in installed and can be imported, but, for some reason, not during installation.

When trying the same with Python 3.11, downgrading pip to 23.0.1 helps, but downgrading to pip 23.1 does not. I couldn't test it with 3.12 where downgrading pip to 23.0.1 doesn't seem to be supported (AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?).

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux, Windows

[da-woods]

The pip issue tracker is at https://github.com/pypa/pip/issues. This doesn't look like a bug in Python itself.

I suspect this is due to isolated build environments though. It sets up a new empty environment to build your package in so you need to make sure your package correctly declares its dependencies.

[Zabolekar]

@da-woods ok, thanks, will report it there.