New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 38.5.0 fails to build extension #1271

Closed
djhoese opened this Issue Feb 5, 2018 · 10 comments

Comments

Projects
None yet
4 participants
@djhoese

djhoese commented Feb 5, 2018

I first noticed this on a travis build for one of my projects. NetCDF4-python (one of my dependencies) fails to build when setuptools 38.5.0 is installed. Forcing the version to 38.4.0 fixes it. See https://travis-ci.org/pytroll/satpy/jobs/337509200#L940 for the failed job which ends with this exception:

      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/opt/python/3.6.3/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/opt/python/3.6.3/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 133, in finalize_options
        _build_ext.finalize_options(self)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Distutils/build_ext.py", line 19, in finalize_options
        self.distribution.ext_modules)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 920, in cythonize
        aliases=aliases)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 815, in create_extension_list
        kwds = deps.distutils_info(file, aliases, base).values
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 670, in distutils_info
        return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 680, in transitive_merge
        node, extract, merge, seen, {}, self.cimported_files)[0]
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 685, in transitive_merge_helper
        deps = extract(node)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 650, in distutils_info0
        cimports, externs, incdirs = self.cimports_externs_incdirs(filename)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Utils.py", line 44, in wrapper
        res = cache[args] = f(self, *args)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 555, in cimports_externs_incdirs
        for include in self.included_files(filename):
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Utils.py", line 44, in wrapper
        res = cache[args] = f(self, *args)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 537, in included_files
        include_path = self.context.find_include_file(include, None)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Compiler/Main.py", line 274, in find_include_file
        error(pos, "'%s' not found" % filename)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Compiler/Errors.py", line 178, in error
        raise InternalError(message)
    Cython.Compiler.Errors.InternalError: Internal compiler error: 'constants.pyx' not found

See here for a working build where I forced setuptools to 38.4.0: https://travis-ci.org/pytroll/satpy/jobs/337598254#L863

I'm hoping that this is enough to get someones attention that knows what's wrong, but if a smaller example is needed I can try to work that out.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 5, 2018

I can reproduce. It looks like the issue is caused by 038baa1. The switch to new_build_ext is screwing with the build includes for some reasons: even though include is added to the extension's include_dirs, constants.pyx/netCDF4.pxi are not found. It works after patching netCDF4/_netCDF4.pyx like this:

@@ -1017,8 +1017,8 @@ from numpy import ma
 from libc.string cimport memcpy, memset
 from libc.stdlib cimport malloc, free
 import_array()
-include "constants.pyx"
-include "netCDF4.pxi"
+include "include/constants.pyx"
+include "include/netCDF4.pxi"
 IF HAS_NC_PAR:
     cimport mpi4py.MPI as MPI
     from mpi4py.libmpi cimport MPI_Comm, MPI_Info, MPI_Comm_dup, MPI_Info_dup, \

IMHO netCDF4' setup.py probably needs to be updated and the bug should be reported to them.

@jaraco

This comment has been minimized.

Member

jaraco commented Feb 5, 2018

That commit references #1270.

My expectation was that new_build_ext was a compatible drop in. Since it's not, I'm tempted to roll back that change until we can get a better understanding of what compatibility concerns might be. I'd like get a reading from the Cython project maintainers on what they recommend for a project such as netCDF4.

@scoder: Can you comment - is the switch from build_ext to new_build_ext something that's expected to have compatibility concerns? Is there a way that either the Cython-based projects or setuptools can provide compatibility behavior to smooth the transition?

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 5, 2018

It does look like it's not compatible: see Cython's changelog.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 6, 2018

It looks like it also breaks the build_ext command:

running build_ext
Traceback (most recent call last):
  File "./setup.py", line 389, in <module>
    cmdclass=cmdclass,
  File ".../setuptools/setuptools/__init__.py", line 129, in setup                   
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup                                        
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands                                 
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 973, in run_command                                  
    cmd_obj.ensure_finalized()
  File "/usr/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized                              
    self.finalize_options()
  File ".../setuptools/setuptools/command/build_ext.py", line 133, in finalize_options
    _build_ext.finalize_options(self)
  File "~/.local/lib/python3.6/site-packages/Cython/Distutils/build_ext.py", line 20, in finalize_options
    super(build_ext, self).finalize_options()
TypeError: super(type, obj): obj must be an instance or subtype of type

Which in turns break the develop command and editable installs with pip (that's with a project that does not even build extensions).

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 6, 2018

Nevermind, that last issue is a bug in Cython 0.25.2, it works fine with 0.27.3.

@vallsv

This comment has been minimized.

Contributor

vallsv commented Feb 6, 2018

Thanks for the rollback. We had the same problem here.

The new_build_ext from cython was building the extensions at the end of the finalize_options, instead of at the build_extensions stage. As result we was not able to configure anything on our side before the cythonization.

I really think it is better to build the cython extensions at the build stage, but it is really a matter of point of view.

If you still want to use new_build_ext, please create a hook method, or wrap the cythonization into a method, like this we still can have some way to patch the cythonization stage.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 6, 2018

@vallsv: just to be clear the change has not been (yet?) reverted.

@vallsv

This comment has been minimized.

Contributor

vallsv commented Feb 6, 2018

Oh sorry there is still no rollback :-)

Then for more explanation, on our projects (https://github.com/silx-kit/pyFAI) we are cythonizing files by ourselves to have a little more features (manage cython cache, and portable openmp options).

I try to move our code from build_extension to finalize_options, to match setuptools 38.5.0 changes but it is not working cause at this time we don't have information on the compiler (self.compiler). That is why, on our side, it would be very helpful if setuptools could provide a way to patch the build_ext cythonization stage in some clean way.

@djhoese

This comment has been minimized.

djhoese commented Feb 6, 2018

With the revert and installing 38.5.1 travis tests are now passing again. Thanks.

@vallsv

This comment has been minimized.

Contributor

vallsv commented Feb 7, 2018

Same here. Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment