New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quick Test Routine After Import #2061

Closed
ax3l opened this Issue Jan 8, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@ax3l

ax3l commented Jan 8, 2018

Hi,

we are currently facing an interesting issue downstream in setuptools with an imported, incompatible Cython install that might exist in a user's environment, e.g. while working with python virtual environments, conda or spack: pypa/setuptools#1229 (comment)

The main issue comes down to the fact that a downstream "optional usage of Cython" in the form of

try:
    # Attempt to use Cython for building extensions, if available
    from Cython.Distutils.build_ext import build_ext as _build_ext
except ImportError:
    # some fallback such as
    _build_ext = _du_build_ext

will not catch any errors that can occur when using a compiled library such as Cython. Exemplary errors can look like

failed to import Cython: /home/axel/.local/lib/python2.7/site-packages/Cython/Compiler/Scanning.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
Unexpected error: <class 'distutils.errors.DistutilsPlatformError'>

in the case of symbol mismatches between currently used virtual environment, system or user libraries. Unfortunately, such errors are not triggered on import but as soon as the imported functionality is used later on. (A full reproducer is posted here including a Dockerfile.)

We were wondering if there already is or if one could add a short, side-effect free "test" interface that one could call directly after the import in the same try scope in order to reliably trigger and catch incompatibly imported Cython installs?

Any feedback or suggestions are very welcome!

@scoder

This comment has been minimized.

Show comment
Hide comment
@scoder

scoder Jan 19, 2018

Contributor

Three things:

  • It's intentional that importing the build_ext module does not import the compiler.
  • It would be better to import new_build_ext instead of build_ext. It's much closer to what people would expect as a default behaviour these days, since it calls cythonize() internally.
  • You can just import the compiler explicitly to detect errors like the one you mentioned. Just import Cython.Compiler.Main or from Cython.Build import cythonize or something like that. Drawback is, obviously, that all of Cython gets imported early even if it's not used at all.
Contributor

scoder commented Jan 19, 2018

Three things:

  • It's intentional that importing the build_ext module does not import the compiler.
  • It would be better to import new_build_ext instead of build_ext. It's much closer to what people would expect as a default behaviour these days, since it calls cythonize() internally.
  • You can just import the compiler explicitly to detect errors like the one you mentioned. Just import Cython.Compiler.Main or from Cython.Build import cythonize or something like that. Drawback is, obviously, that all of Cython gets imported early even if it's not used at all.
@ax3l

This comment has been minimized.

Show comment
Hide comment
@ax3l

ax3l Jan 20, 2018

@jaraco would one of the above solutions be suitable for you, downstream in setuptools?

ax3l commented Jan 20, 2018

@jaraco would one of the above solutions be suitable for you, downstream in setuptools?

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Jan 20, 2018

The third option looks like it may be the officially-supported way to detect if the compiler is available. Probably just __import__('Cython.Compiler.Main') with a comment explaining why.

jaraco commented Jan 20, 2018

The third option looks like it may be the officially-supported way to detect if the compiler is available. Probably just __import__('Cython.Compiler.Main') with a comment explaining why.

@scoder

This comment has been minimized.

Show comment
Hide comment
@scoder

scoder May 27, 2018

Contributor

Closing, as I think an explicit import solves this issue.

Contributor

scoder commented May 27, 2018

Closing, as I think an explicit import solves this issue.

@scoder scoder closed this May 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment