Symbol not found: __PyThreadState_UncheckedGet for compiled Cython, Python 3.5

[matthew-brett]

There is a new error after the Cython 0.27 release, found here:

https://travis-ci.org/MacPython/h5py-wheels/jobs/278928715

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-66_llhko-build/setup.py", line 167, in <module>
      cmdclass = CMDCLASS,
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 204, in run
      self.run_command('build')
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-66_llhko-build/setup_build.py", line 149, in run
      from Cython.Build import cythonize
    File "/Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Build/__init__.py", line 1, in <module>
      from .Dependencies import cythonize
    File "/Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 59, in <module>
      from ..Compiler.Main import Context, CompilationOptions, default_options
    File "/Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Compiler/Main.py", line 28, in <module>
      from .Scanning import PyrexScanner, FileSourceDescriptor
  ImportError: dlopen(/Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Compiler/Scanning.cpython-35m-darwin.so, 2): Symbol not found: __PyThreadState_UncheckedGet
    Referenced from: /Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Compiler/Scanning.cpython-35m-darwin.so
    Expected in: flat namespace
   in /Users/travis/build/MacPython/h5py-wheels/venv/lib/python3.5/site-packages/Cython/Compiler/Scanning.cpython-35m-darwin.s

Testing locally this error does not occur when Cython is installed via:

pip install . --install-option="--no-cython-compile"

It does occur when installing without the --no-cython-compile module.

Any ideas what might be going on?

[matthew-brett]

Possibly relevant : pybind/pybind11#314 and https://bugs.python.org/issue26154

[matthew-brett]

The failure is only for Python 3.5 on OSX. The compilation there is for Python 3.5.1. I think the Cython wheels are compiling against Python 3.5.3. Am I right in thinking that

cython/Cython/Utility/Exceptions.c

Line 16 in 7cac3e0

#define __Pyx_PyThreadState_assign $local_tstate_cname = _PyThreadState_UncheckedGet();

will pull in that symbol when compiling on any Python 3.5, but it will not be defined when running on Python 3.5 < 3.5.2?

[matthew-brett]

Sorry - to clarify, I think it's only failing on OSX, because on Manylinux, we are testing with Python 3.5.3, but with OSX we are testing with Python 3.5.1 (in the h5py case).

[scoder]

Thank you for your analysis.

[eugene-yang]

Is there any update about this issue?
I used Python 3.7 with Cython 0.29.6 in OSX and still have the same problem.

[scoder]

The issue is resolved since 2017 (Cython 0.27.1) and it was never a problem with Python 3.6 or later. Please make sure you are correctly generating your .c files (look at the first like to see the Cython version that was used).

If you can provide information that suggests why there would still be a problem, please bring it up.