Allow setup.py/pip to run query commands without needing numpy.

[evertrol]

pip downloads and queries packages first, before trying to build them. In a clean environment, numpy is missing and any query therefore fails, though the build could actually proceed (provided numpy is built and installed first).
A conditional import depending on the setup command used can prevent pip from failing on the installation.

In particular, this can be an issue when tox is used for automated testing: tox sets up a virtualenv without numpy, uses pip to download and query netCDF4 (python setup.py egg_info) and then fails during that inquiry since numpy is required, even if numpy is listed first in tox.ini and will be installed before netCDF4.

Note: the current solution still poses problems when running python setup.py manually and using an invalid command: instead of letting distutils/setuptools point out the invalid command to the user, an ImportError is raised for a missing numpy.

[jswhit]

Since I don't have a system without numpy, I can't test this. Is the main idea to have pip query setup.py to find the dependencies without failing on the numpy import? Obviously, netcdf4-python cannot be build before numpy is installed.

[shoyer]

The easy way to test not having numpy on a computer where you have it installed is to use a virtual environment:

>>> virtualenv test-no-numpy
Using real prefix '/System/Library/Frameworks/Python.framework/Versions/2.7'
Cannot find file /export/disk0/wb/python2.6/lib/python2.7/readline.so (bad symlink)
New python executable in test-no-numpy/bin/python
Installing setuptools, pip...done.
>>> . test-no-numpy/bin/activate
>>> pip list
netCDF4 (1.1.0, /Users/shoyer/dev/netcdf4-python)
pip (1.5.4)
setuptools (2.2)
wsgiref (0.1.2)
>>> git branch
* master
>>> python setup.py --name
Traceback (most recent call last):
  File "setup.py", line 1, in <module>
    import os, sys, subprocess, numpy, shutil
ImportError: No module named numpy
>>> git checkout setup-pip-numpy-requirement
Branch setup-pip-numpy-requirement set up to track remote branch setup-pip-numpy-requirement from evertrol by rebasing.
Switched to a new branch 'setup-pip-numpy-requirement'
>>> python setup.py --name

HDF5_DIR environment variable not set, checking some standard locations ..
checking /Users/shoyer ...
checking /usr/local ...
HDF5 found in /usr/local

NETCDF4_DIR environment variable not set, checking standard locations..
checking /Users/shoyer ...
checking /usr/local ...
netCDF4 found in /usr/local
using netcdf library version 4.3.0
netCDF4

So yes, in principle the idea is to be able to lookup dependencies without failing on the numpy import. To get that working, I think you'll also need to add requires=['numpy'] to the setup function in setup.py. Of course, that only checks the python dependencies... something like conda is a fuller solution to the problem but I think this is a nice step in the right direction.

[evertrol]

Thanks.

Yes, it is indeed in virtualenvs (what tox uses to set up a clean testing environment) where things fail; most OSes come with numpy installed by default, so outside a virtualenv, it's rare to run into this.

requires=['numpy'] seems logical to implement (there's also setup_requires in setuptools).
For the record, I list some other packages where this problem occurred and was solved:

• scipy: Fixed #1429: "pip install numpy scipy" was failing scipy/scipy#453
• h5py: Allows setup.py to execute without having NumPy installed beforehand. h5py/h5py#356
• and the original pip issue Do not run egg_info to each package in requirements list before installing the previous packages pypa/pip#25