Numpy dependency install fails if not already installed

[jgosmann]

I tried to install Nengo in a virtualenv without installing NumPy beforehand. The Nengo seemed to try to install NumPy, but failed with some compile error. Installing NumPy with pip first and then Nengo worked.

I have to look further into what exactly the error was etc. but don't have time right now. So this ticket is for now just a reminder for me.

[Seanny123]

I have the same problem, here is the error message:

Could not locate executable gfortran
Could not locate executable f95
Could not locate executable ifort
Could not locate executable ifc
Could not locate executable lf95
Could not locate executable pgfortran
Could not locate executable f90
Could not locate executable f77
Could not locate executable fort
Could not locate executable efort
Could not locate executable efc
Could not locate executable g77
Could not locate executable g95
Could not locate executable pathf95
don't know how to compile Fortran code on platform 'posix'
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
 int exp (void);
     ^
_configtest.o: In function `main':
/tmp/easy_install-SOqQ4y/numpy-1.9.1/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.o: In function `main':
/tmp/easy_install-SOqQ4y/numpy-1.9.1/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
 int exp (void);
     ^
_configtest.c:1:20: fatal error: Python.h: No such file or directory
 #include <Python.h>
                    ^
compilation terminated.
_configtest.c:1:20: fatal error: Python.h: No such file or directory
 #include <Python.h>
                    ^
compilation terminated.
Traceback (most recent call last):
  File "setup.py", line 81, in <module>
    'tox': Tox,
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 27, in run
    self.install_for_development()
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 129, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 682, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/pkg_resources.py", line 631, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/pkg_resources.py", line 871, in best_match
    return self.obtain(req, installer)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/pkg_resources.py", line 883, in obtain
    return installer(requirement)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 595, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 625, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 822, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1028, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1013, in run_setup
    run_setup(setup_script, args)
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 50, in run_setup
    lambda: execfile(
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 100, in run
    return func()
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 52, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 251, in <module>

  File "setup.py", line 243, in setup_package

  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/core.py", line 169, in setup
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/saubin/.virtualenvs/nengo/local/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 148, in run
    self.run_command("egg_info")
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/command/egg_info.py", line 10, in run
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/command/build_src.py", line 153, in run
  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/command/build_src.py", line 170, in build_sources
  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/command/build_src.py", line 329, in build_extension_sources
  File "/tmp/easy_install-SOqQ4y/numpy-1.9.1/numpy/distutils/command/build_src.py", line 386, in generate_sources
  File "numpy/core/setup.py", line 432, in generate_config_h

  File "numpy/core/setup.py", line 42, in check_types
    TestCommand.finalize_options(self)
  File "numpy/core/setup.py", line 293, in check_types

SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

However, unlike @jgosmann, installing numpy with pip install numpy did not remedy the problem for me.

[tbekolay]

Are you able to pip install numpy by itself? If not, then:

sudo apt-get install python-dev build-essential

and try again. This will at least let you install it, but note that this Numpy install will be pretty slow. @hunse has instructions on a faster Numpy install using openblas somewhere...

[Seanny123]

Sorry about that, I apparently borked my install by playing around with Anaconda while trying to get the offline install to work. Installing numpy using pip install numpy now works and solves the problem, however I get the same error if I don't install numpy.

Is the fact that numpy is giving us such a hassle an indication that we should be using conda as opposed to setup-tools? I ask as a noob to this whole python-building business.

[tbekolay]

We can certainly write a conda recipe for Nengo, but we can't really get around writing a setup.py that works with setuptools, as that's necessary for PyPI. The Python-building business is really a mess right now, unfortunately; people are trying to make it easier but it's still butts.

[Seanny123]

Confirmed existence on Ubuntu 14.04 clean install in #528.

[jgosmann]

Relevant, but unhelpful SO thread.
There seems to be no easy way to obtain at least some basic debug info (like the exact compiler command invoked). :/

[jgosmann]

Another unanswered SO thread

[jgosmann]

Apparently, this is a known but unfixed issue. See numpy/numpy#2434.

[jgosmann]

Some people say install_requires should be avoided.

[tbekolay]

From that thread and previous experience, it seems as though Matplotlib and Pandas have solved this issue (confirmed by making a new virtualenv and pip installing each; worked with MPL and Pandas but not Nengo). We'll have to look into their setup.py scripts and see how they did it; MPL, Pandas.

One other question: if you cd to the nengo directory and do pip install . does it work?

[jgosmann]

pip install and python setup.py install are not the same if what I read is correct. It seems that pip is not affected by that bug.

pip install . seems to work for me (don't have the time right now to wait for the compilation to finish), python setup.py install does not.

[Seanny123]

I investigated Matplotlib, but they literally don't seem to be doing anything different since they end up with the same setup() file that we have for Numpy. Same thing with Pandas. I think may be over my head here.

Given that I'm stuck, I started looking at other build options and I thought I would round them up here:

• conda/anaconda/miniconda: adds a dependency and maintains packages seperate from the system packages, so not ideal if the user already has Python installed according to @tbekolay
• pip/wheels: I can't find the numpy wheels for 32-bit systems and building them ourselves seems excessively difficult
• buildout: this is apparently what SciPy uses, unfortunately from a speed-read of the tutorial the system doesn't look that intuitive. Note to self to check out this PR as well.

Let me know your thoughts on this.

[jgosmann]

I can confirm that pip install . works.

[Seanny123]

Right, I was getting a little bit carried away with my previous comment, as I accidentally expanded the scope of the comment thread from "how do we install numpy" to "how do we package our application".

Focusing back on installing numpy, I talked to Trevor and I'm going to figure out how to install numpy in setup.py using get_pip.py and just opening a subprocess.

[jgosmann]

Any progress on this?

[tbekolay]

OK... I think I fixed it, but honestly it's a weird fix (short though). For some reason, NumPy installs fine if it's part of setup_requires, just not install_requires. And, despite some things I read online, doing this only installs NumPy once, so I think it's good enough (most people will install nengo with pip install nengo anyhow). I'll make the PR shortly; would appreciate testers on various platforms (I'm on Mac OS X right now).

edit: PR made: #568.

[jgosmann]

Works for me on Linux.

[jgosmann]

Merged PR #568

[swedishhh]

tbekolay commented on 29 Nov 2014

Thank you!