How to Ignore Cython, even if found #1229
That is not in all cases ideal, e.g. in a package manager (such as spack) or a virtual python environment that does not have a cython installed, while at the same time a system-wide Cython exists. In such a setup, nearly always binary incompatibilities with the system-wide python install will occur.
Is there any way to express "try not to import cython (even when found)" to setuptools?
The text was updated successfully, but these errors were encountered:
Can you expand a little on
I'd assumed that the Cython
How is it that builds are affected by the presence of Cython?
The problem I am facing is that
try: # Attempt to use Cython for building extensions, if available from Cython.Distutils.build_ext import build_ext as _build_ext
succeeds but a later call to
from setuptools.command.build_ext import build_ext as BuildExtCommand class BuildExtraLibraries(BuildExtCommand): def run(self): return BuildExtCommand.run(self)
instead of something from the python I am currently working with (which has no cython package installed).
I am trying to build a little, not too complex reproducer but it's a bit complex in this situation (need to install two incompatible python versions, etc.).
All right, I have a demonstrating
A default CPython 2.7 build with some version of gcc (e.g. 5) and building some version of cython.
Virtual / Per-User Software
A conda provided CPython 2.7 build with some other version of gcc (4.8) and without cython.
The problem now is, that setuptools will pick on
This effect is commonly known as side-effect during install of e.g. C/C++ libraries. Instead of a "pick any" of a dependency an option to disable/pick-a-specific cython would be needed for better control. (e.g. modern CMake scripts usually expose an "AUTO/ON/OFF" option to users [example]).
Long story short: as soon as you do a
try: import Cython
inside setuptools, Cython needs to be considered a public, optional dependency of setuptools and build systems / package managers / non-isolated virtual environments need the control to explicitly set the path to it or to disable it fully even if an install exists (in e.g.
docker build -t setuptools .
# updated: forgot to copy-paste the base image I used :) FROM ubuntu:16.04 # general environment for docker ENV DEBIAN_FRONTEND=noninteractive \ FORCE_UNSAFE_CONFIGURE=1 RUN apt-get update \ && apt-get install -y --no-install-recommends \ bzip2 \ build-essential \ pkg-config \ libc6-dev \ python-minimal \ python-dev \ python-pip \ ca-certificates \ poppler-utils \ coreutils \ curl \ git \ lsb-release \ wget \ && rm -rf /var/lib/apt/lists/* # this is the "SYSTEM-WIDE" install of python and cython RUN pip install setuptools --user --no-binary :all: RUN pip install cython --user --no-binary :all: RUN python --version && which python RUN /root/.local/bin/cython --version # here starts the "USER-SCOPE": either in $HOME, conda, a virtual env, or e.g. spack RUN wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh && \ chmod +x miniconda.sh && \ ./miniconda.sh -b && \ ls RUN git clone --depth 10 https://github.com/matplotlib/matplotlib.git RUN /root/miniconda2/bin/conda create -y -p /tmp/py27 python=2.7 && \ /bin/bash -l -c "source /root/miniconda2/bin/activate /tmp/py27 && \ /root/miniconda2/bin/conda install -y gcc libgcc=4.8.5 numpy pkgconfig libpng freetype && \ cd matplotlib && \ which python && \ python setup.py build && \ python setup.py install"
No, I do have a cython installed - system wide. And it should not be picked inside a virtual env or conda env or spack env that is binary incompatible. The problem is, that all three are not isolating the user, so a "try to pick any cython" strategy in setuptools has this very confusing side-effect for users inside a virtual env/conda/spack.
As a quick work-around one could do e.g. the following: inside the
from distutils.command.build_ext import build_ext as _du_build_ext try: # Attempt to use Cython for building extensions, if available from Cython.Distutils.build_ext import build_ext as _build_ext # Test if _build_ext is usable/compatible class TestBuildExt(_build_ext): def run(self): return _build_ext.run(self) TestBuildExt().run() # not sure if some cleanup will be needed now except ImportError, distutils.errors.DistutilsPlatformError: _build_ext = _du_build_ext
This will still not expose (imho needed) control over the use of Cython's
Thanks @ax3l for the reproducer. I've uploaded it as this gist. I then made a couple of revisions. The first was to specify a base image, which I suspect was necessary because I'm testing on macOS. At that point, I was able to run the build and get the error you described.
But I noticed one thing - that you're not in fact installing Cython system-wide; you're installing it in the user site (
In the next revision, I installed Cython system-wide, and the issue goes away. Now that Cython is installed only to the system Python 2.7, it no longer appears in the miniconda install of Python 2.7, and all is well.
In rev 4, I took another tack and disabled that user-local environment before invoking the build. This also bypasses the issue. One could alternatively have set
I think this demonstrates that the issue originates from an incorrect usage of user-local environments and presents a couple of plausible workarounds.
I do sympathize with the issue and thinking about the presented workaround in Setuptools, I'm reluctant to go with that approach because of the complexity of behavior it adds on import. If on the other hand cython has or could add a call to check that it's viable in this environment, like
try: # Attempt to use Cython for building extensions, if available from Cython.Distutils.build_ext import build_ext as _build_ext __import__('cython').assert_viable() except (ImportError, AssertionError): _build_ext = _du_build_ext
In other words, if Cython is willing to support this use-case, then setuptools will support it as well.
Thank you for your detailed answer!
Ups, sorry - copy & paste omitted on my side. Updated the original
yes, absolutely. My wording was sloppy, I just meant "it's somewhere outside (of virtual env/conda/...) and can be found".
Same for me, it then can not be found anymore as expected.
interesting, didn't know that option exists. looks like a very good candidate for a package manager like spack! I wonder why conda is not using this by default as well (on activate).
I agree that this is probably not intended but very, very common. Users start with some environment they have when creating a new virtual environment or installing conda. I've not yet seen a manual recommending "delete your
Absolutely, this would also be a great-solution if
Orthogonal: Maybe also enable users to even skip the
Upstream we got the recommendation that we should just include