Including bottleneck as dependency in install_requires in setup.py causes installation to fail

[drdavella]

Here's a minimal test case. Use the setup.py below and then run python setup.py install or python setup.py develop in a clean environment (i.e. one that doesn't already have numpy installed):

from setuptools import setup

setup(
    install_requires=['bottleneck'],
)

The full failure is listed below. A workaround is to add numpy to setup_requires, but it seems like downstream packages should not be required to do this. This is probably not noticed by most users since numpy is already installed in most environments, but the current behavior is a bug.

Searching for bottleneck
Reading https://pypi.org/simple/bottleneck/
Downloading https://files.pythonhosted.org/packages/05/ae/cedf5323f398ab4e4ff92d6c431a3e1c6a186f9b41ab3e8258dff786a290/Bottleneck-1.2.1.tar.gz#sha256=6efcde5f830aed64feafca0359b51db0e184c72af8ba6675b4a99f263922eb36
Best match: Bottleneck 1.2.1
Processing Bottleneck-1.2.1.tar.gz
Writing /var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/setup.cfg
Running Bottleneck-1.2.1/setup.py -q bdist_egg --dist-dir /var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/egg-dist-tmp-q57bo93i
No Bottleneck unit testing available.
Traceback (most recent call last):
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/setup.py", line 110, in <module>
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 163, in run
    self.run_command("egg_info")
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 295, in run
    self.find_sources()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 302, in find_sources
    mm.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 533, in run
    self.add_defaults()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 569, in add_defaults
    sdist.add_defaults(self)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/command/sdist.py", line 228, in add_defaults
    self._add_defaults_ext()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/command/sdist.py", line 311, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 299, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/setup.py", line 24, in finalize_options
AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./setup.py", line 6, in <module>
    install_requires=['bottleneck'],
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/develop.py", line 154, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 749, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/pkg_resources/__init__.py", line 777, in resolve
    replace_conflicting=replace_conflicting
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1060, in best_match
    return self.obtain(req, installer)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1072, in obtain
    return installer(requirement)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 676, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 702, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 887, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1155, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1141, in run_setup
    run_setup(setup_script, args)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/setup.py", line 110, in <module>
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 163, in run
    self.run_command("egg_info")
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 295, in run
    self.find_sources()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 302, in find_sources
    mm.run()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 533, in run
    self.add_defaults()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 569, in add_defaults
    sdist.add_defaults(self)
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/command/sdist.py", line 228, in add_defaults
    self._add_defaults_ext()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/command/sdist.py", line 311, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 299, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/Users/ddavella/miniconda3/envs/bottleneck-dep/lib/python3.7/distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/var/folders/6s/st61dq157dnd1nwd56hzy16h000153/T/easy_install-_6ubag8s/Bottleneck-1.2.1/setup.py", line 24, in finalize_options
AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'

[drdavella]

Are there any plans to fix this? This continues to affect multiple developers on my team since we have an indirect dependency on this package.

[kwgoodman]

Can you have your team test your proposed solution with and without numpy preinstalled? If it work make a PR

[drdavella]

I proposed a workaround for downstream packages, but not a solution.

Basically it's not safe to import numpy in setup.py since you can't assume that it's already installed in an arbitrary environment.

I think the right long-term solution here is probably PEP 518. It requires downstream users to have pip-10.0 or later, though, which can't be guaranteed at this point.

[kwgoodman]

All this packaging stuff is a mystery to me. You ask if I have plans to fix it but you don't have a solution? Is that the status?

[drdavella]

I think the right long-term solution here is probably PEP 518.

If every bug report came with a solution for free, my job would be a lot easier. If you don't care to address this, feel free to close. I simply asked about the status since there had been no reply to my original report.

[kwgoodman]

I wasn't expecting a solution for free. I was just confused about the status---whether or not there was a proposed solution.

[drdavella]

Sorry, I misunderstood 🤐.

I can try to open a PR soon that adds a pyproject.toml but admittedly it's going to be bit hard to test. I realize this seems fairly minor, but it's been quite an annoyance, especially since we don't depend on this project directly (although it seems very cool).

To be fair, Python packaging is a mess. It seems like PEP 518 offers a sane way forward, but unfortunately it's going to be awhile before it's safe to assume that everyone has a sufficiently recent version of pip. In the meantime, we're stuck with problems like this one.

The other workaround solution (which would be a pain) is to make your setup.py automatically install numpy and then refresh the imports so that it's guaranteed to be installed in time. I don't necessarily recommend it, but this is what astropy does in order to ensure that numpy is installed prior to building extensions that depend on it. In theory, you could use astropy_helpers, which was designed for this purpose, but again I'm not sure I actually recommend taking that path.

[kwgoodman]

So if you added pyproject.toml then everyone with pip 10 or greater would not have this problem? And older pips would ignore it?

[drdavella]

Yes, that's how I understand it.

[astrofrog]

Just for the record, the ideal solution is indeed the pyproject.toml file (and not setup_requires) - see http://numpy-discussion.10968.n7.nabble.com/Issue-with-setup-requires-and-1-16-release-candidates-td46600.html for more details.

[drdavella]

The issue reported here is addressed by #203. However, I realized that the specific case I described in this issue (using setup.py install/develop) may never actually work properly, even with the addition of pyproject.toml. This is because the community is moving away from the use of setup.py as an entry point, even for installation. So the correct thing to do going forward will be pip install [-e] ., which will take advantage of pyproject.toml.