extensions fail to build when dependency of package with extensions, and Cython installed #723

johnyf · 2016-01-05T21:59:59Z

The following minimal example, with foo.c trivial (and trivial package hoho content) fails to install with python setup.py install, due to psutil failing to compile its extensions.

(More accurately, hoho is installed, but setuptools fails in its attempt to install the dependency psutil after finishing with hoho itself).

from setuptools import setup, Extension

name = 'hoho'
version = '0.0.1'
ext_modules = [Extension('foo', ['hoho/foo.c'])]
setup(
     name=name,
     version=version,
     packages=[name],
     package_dir={name: name},
     install_requires=['psutil'],
     ext_modules=ext_modules)

The failure happens only if cython is installed. If no ext_modules are passed, then no error arises.
The output is:

(bug)hoho:pip install cython
Collecting cython
Installing collected packages: cython
Successfully installed cython-0.23.4
[154839 refs]
(bug)hoho:python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to hoho.egg-info/requires.txt
writing hoho.egg-info/PKG-INFO
writing top-level names to hoho.egg-info/top_level.txt
writing dependency_links to hoho.egg-info/dependency_links.txt
reading manifest file 'hoho.egg-info/SOURCES.txt'
writing manifest file 'hoho.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.4-x86_64/egg
running install_lib
running build_py
running build_ext
creating build/bdist.macosx-10.4-x86_64/egg
copying build/lib.macosx-10.4-x86_64-2.7-pydebug/foo.so -> build/bdist.macosx-10.4-x86_64/egg
creating build/bdist.macosx-10.4-x86_64/egg/hoho
copying build/lib.macosx-10.4-x86_64-2.7-pydebug/hoho/__init__.py -> build/bdist.macosx-10.4-x86_64/egg/hoho
copying build/lib.macosx-10.4-x86_64-2.7-pydebug/hoho/hehe.py -> build/bdist.macosx-10.4-x86_64/egg/hoho
byte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/hehe.py to hehe.pyc
creating stub loader for foo.so
byte-compiling build/bdist.macosx-10.4-x86_64/egg/foo.py to foo.pyc
creating build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/PKG-INFO -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/SOURCES.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/dependency_links.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/requires.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/top_level.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
writing build/bdist.macosx-10.4-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating 'dist/hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg' and adding 'build/bdist.macosx-10.4-x86_64/egg' to it
removing 'build/bdist.macosx-10.4-x86_64/egg' (and everything under it)
Processing hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg
Copying hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages
Adding hoho 0.0.1 to easy-install.pth file

Installed /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg
Processing dependencies for hoho==0.0.1
Searching for psutil
Reading https://pypi.python.org/simple/psutil/
Best match: psutil 3.3.0
Downloading https://pypi.python.org/packages/source/p/psutil/psutil-3.3.0.tar.gz#md5=0625db01ea46417308db94cc59362b65
Processing psutil-3.3.0.tar.gz
Writing /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.cfg
Running psutil-3.3.0/setup.py -q bdist_egg --dist-dir /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/egg-dist-tmp-MmWzk4
Traceback (most recent call last):
  File "setup.py", line 12, in <module>
    ext_modules=ext_modules)
  File "/Users/ifilippi/python/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/ifilippi/python/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/ifilippi/python/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 390, in run
    self.easy_install(spec, not self.no_deps)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 620, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 671, in install_item
    self.process_distribution(spec, dist, deps)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 719, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py", line 846, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1091, in best_match
    return self.obtain(req, installer)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1103, in obtain
    return installer(requirement)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 639, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 669, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 852, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1080, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1066, in run_setup
    run_setup(setup_script, args)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 242, in run_setup
    raise
  File "/Users/ifilippi/python/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Users/ifilippi/python/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
    compat.reraise(type, exc, self._tb)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 239, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 269, in run
    return func()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 238, in runner
    _execfile(setup_script, ns)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.py", line 268, in <module>

  File "/var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.py", line 265, in main

  File "/Users/ifilippi/python/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/ifilippi/python/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/ifilippi/python/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 151, in run
    self.run_command("egg_info")
  File "/Users/ifilippi/python/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/ifilippi/python/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 185, in run
    self.find_sources()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 208, in find_sources
    mm.run()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 292, in run
    self.add_defaults()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 321, in add_defaults
    sdist.add_defaults(self)
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/sdist.py", line 131, in add_defaults
    build_ext = self.get_finalized_command('build_ext')
  File "/Users/ifilippi/python/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/Users/ifilippi/python/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 97, in finalize_options
    self.check_extensions_list(self.extensions)
  File "/Users/ifilippi/python/lib/python2.7/distutils/command/build_ext.py", line 362, in check_extensions_list
    ("each element of 'ext_modules' option must be an "
setuptools.sandbox.UnpickleableException: DistutilsSetupError("each element of 'ext_modules' option must be an Extension instance or 2-tuple",)
[223031 refs]

It appears to be an instance of:

issue 309 of setuptools
the discussion over here: http://stackoverflow.com/questions/21594925/error-each-element-of-ext-modules-option-must-be-an-extension-instance-or-2-t

and possibly related to:

It arose when testing dd on Travis.
There is no other workaround that I expect will work for dd, other than to either:

avoid this from happening in psutil
pursue a fix in setuptools or cython (not sure whose the culprit -- could be distribute, but not expected to make a fix available soon)
drop psutil from install_requires of dd.

For comparison, a successful installation produces as output:

(bug)hoho:python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to hoho.egg-info/requires.txt
writing hoho.egg-info/PKG-INFO
writing top-level names to hoho.egg-info/top_level.txt
writing dependency_links to hoho.egg-info/dependency_links.txt
reading manifest file 'hoho.egg-info/SOURCES.txt'
writing manifest file 'hoho.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.4-x86_64/egg
running install_lib
running build_py
creating build/lib
creating build/lib/hoho
copying hoho/__init__.py -> build/lib/hoho
copying hoho/hehe.py -> build/lib/hoho
creating build/bdist.macosx-10.4-x86_64/egg
creating build/bdist.macosx-10.4-x86_64/egg/hoho
copying build/lib/hoho/__init__.py -> build/bdist.macosx-10.4-x86_64/egg/hoho
copying build/lib/hoho/hehe.py -> build/bdist.macosx-10.4-x86_64/egg/hoho
byte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/hehe.py to hehe.pyc
creating build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/PKG-INFO -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/SOURCES.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/dependency_links.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/requires.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
copying hoho.egg-info/top_level.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/hoho-0.0.1-py2.7.egg' and adding 'build/bdist.macosx-10.4-x86_64/egg' to it
removing 'build/bdist.macosx-10.4-x86_64/egg' (and everything under it)
Processing hoho-0.0.1-py2.7.egg
Copying hoho-0.0.1-py2.7.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages
Adding hoho 0.0.1 to easy-install.pth file

Installed /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/hoho-0.0.1-py2.7.egg
Processing dependencies for hoho==0.0.1
Searching for psutil
Reading https://pypi.python.org/simple/psutil/
Best match: psutil 3.3.0
Downloading https://pypi.python.org/packages/source/p/psutil/psutil-3.3.0.tar.gz#md5=0625db01ea46417308db94cc59362b65
Processing psutil-3.3.0.tar.gz
Writing /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-8JI14G/psutil-3.3.0/setup.cfg
Running psutil-3.3.0/setup.py -q bdist_egg --dist-dir /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-8JI14G/psutil-3.3.0/egg-dist-tmp-pM1MD7
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: manifest_maker: MANIFEST.in, line 18: 'recursive-include' expects <dir> <pattern1> <pattern2> ...

zip_safe flag not set; analyzing archive contents...
psutil.__init__: module references __file__
creating /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg
Extracting psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages
Adding psutil 3.3.0 to easy-install.pth file

Installed /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg
Finished processing dependencies for hoho==0.0.1
[119389 refs]

The text was updated successfully, but these errors were encountered:

johnyf · 2016-01-05T22:11:00Z

An installation with pip does not fail, but it is not of interest, because package-specific command line arguments need to be passed to setup.py. Currently, these arguments will cause a failure with pip too, because they get passed to all dependencies (e.g., to psutil), see pip #1883.

johnyf · 2016-01-05T22:38:01Z

An ugly workaround is to include psutil in setup_requires.

because `python setup.py install --cudd` fails if `cython` is installed [1]. [1] giampaolo/psutil#723

johnyf · 2016-01-06T13:08:45Z

This bug changes behavior between:

setuptools==18.8: some memory leak apparently causes all my RAM to be filled on OS X v10.8.5 with a python 2.7.10+ (built from sources) inside a virtualenv. The leak happens after: Running psutil-3.3.0/setup.py -q bdist_egg --dist-dir /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-Xqhavh/psutil-3.3.0/egg-dist-tmp-sTg_pI and I kill the installation.
setuptools==18.8.1: the error described above is triggered.

johnyf · 2016-01-06T13:14:24Z

The main difference between setuptools 18.8 and 18.8.1 appears to be the memory blow up reported in setuptools issue 440. So, the fix 2461fe779674 in setuptools to avoid infinite recursion appears to uncover another bug.

johnyf · 2016-01-06T17:31:01Z

Both the demo hoho and psutil use setuptools as normal, and do not mention cython. distutils is independent of any patching that happens later. Therefore, setuptools appears to be responsible, so continuing at setuptools #309.

giampaolo · 2016-01-11T09:13:24Z

Can we close this issue?

johnyf · 2016-01-11T16:22:18Z

Yes. There has not been any reply in issue 309 of setuptools yet.

johnyf · 2016-01-25T00:42:56Z

The issue has been fixed as setuptools 488, see also comments in setuptools 309.

to avoid `cython` affecting installation of `psutil` [1, 2, 3]. [1] https://bitbucket.org/pypa/setuptools/issues/488 [2] https://bitbucket.org/pypa/setuptools/issues/309 [3] giampaolo/psutil#723

johnyf added a commit to tulip-control/dd that referenced this issue Jan 6, 2016

REL: rm from psutil from install_requires

af58bf8

because `python setup.py install --cudd` fails if `cython` is installed [1]. [1] giampaolo/psutil#723

johnyf closed this as completed Jan 11, 2016

joleroi mentioned this issue Feb 2, 2016

setup.py develop option broken in master gammapy/gammapy#432

Closed

ghost mentioned this issue Mar 29, 2016

error: each element of 'ext_modules' option must be an Extension instance or 2-tuple pypa/setuptools#309

Open

shntnu added a commit to cytomining/cytominer-database that referenced this issue Apr 20, 2016

Cython / setuptools / ubuntu issue: giampaolo/psutil#723

a626286

zzl0 mentioned this issue Aug 5, 2016

安装报错 douban/dpark#63

Closed

giampaolo added the bug label Nov 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extensions fail to build when dependency of package with extensions, and Cython installed #723

extensions fail to build when dependency of package with extensions, and Cython installed #723

johnyf commented Jan 5, 2016

johnyf commented Jan 5, 2016

johnyf commented Jan 5, 2016

johnyf commented Jan 6, 2016

johnyf commented Jan 6, 2016

johnyf commented Jan 6, 2016

giampaolo commented Jan 11, 2016

johnyf commented Jan 11, 2016

johnyf commented Jan 25, 2016

extensions fail to build when dependency of package with extensions, and Cython installed #723

extensions fail to build when dependency of package with extensions, and Cython installed #723

Comments

johnyf commented Jan 5, 2016

johnyf commented Jan 5, 2016

johnyf commented Jan 5, 2016

johnyf commented Jan 6, 2016

johnyf commented Jan 6, 2016

johnyf commented Jan 6, 2016

giampaolo commented Jan 11, 2016

johnyf commented Jan 11, 2016

johnyf commented Jan 25, 2016