f2py test fail in pip installed numpy-1.10.1 in virtualenv

[joernhees]

running python -c 'import numpy ; numpy.test();' i get the following error about test_f2py:

$ python -c 'import numpy ; numpy.test();'
Running unit tests for numpy
NumPy version 1.10.1
NumPy relaxed strides checking option: True
NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
Python version 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
nose version 1.3.7
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...SK.S.......S...................................................................................clang: warning: -Wl,-rpath=/var/folders/95/hl31m68d6ks1d8txd27wxq000000gn/T/tmpGRMpd5/libbar.so: 'linker' input unused
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S......................................................................................................................S.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E
======================================================================
ERROR: test_scripts.test_f2py
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/decorators.py", line 146, in skipper_func
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/numpy/tests/test_scripts.py", line 68, in test_f2py
    code, stdout, stderr = run_command([f2py_cmd, '-v'])
  File "/usr/local/lib/python2.7/site-packages/numpy/tests/test_scripts.py", line 48, in run_command
    proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

----------------------------------------------------------------------
Ran 5919 tests in 35.478s

FAILED (KNOWNFAIL=3, SKIP=7, errors=1)

[charris]

Ah, OS X. IIRC, pip is not recommended for the mac, homebrew or one of the other third party package managers is preferred.

[joernhees]

hmmm, well pip is much better supported inside virtualenv and usually more up to date than 3rd party managers... previous versions worked like a charm if you installed non-python dependencies with brew... http://joernhees.de/blog/2014/02/25/scientific-python-on-mac-os-x-10-9-with-homebrew/

as all other tests pass like a charm, i actually suspect this to be a test issue rather than a real one: https://github.com/numpy/numpy/blob/master/numpy/tests/test_scripts.py#L67

[pv]

The assumption in the test that f2pyX.Y (or a correct version of it) is in the PATH is often not true on Linux either. The test seems to be testing something that's not guaranteed in general.

[joernhees]

i inserted a print statement after the line linked above:

        f2py_cmd = 'f2py' + basename(sys.executable)[6:]
        print f2py_cmd
        code, stdout, stderr = run_command([f2py_cmd, '-v'])

prints f2py2.7 just before the test fails... the file installed by pip is in /usr/local/bin/f2py, so it's no wonder that trying to call f2py2.7 fails, as it should just be f2py.

[charris]

Hmm, OK.

[joernhees]

i went ahead to test this in a virtualenv on a Ubuntu 14.04 server i have... didn't break, but observe this:

(np)hees@server:~/tmp$ f2py
f2py         f2py2.7      f2py2.7-dbg  f2py3        f2py3.4      f2py3.4-dbg  f2py3-dbg    f2py-dbg
(np)hees@server:~/tmp$ which f2py
/home/hees/tmp/np/bin/f2py
(np)hees@server:~/tmp$ which f2py2.7
/usr/bin/f2py2.7

it seems as if the system-wide installed python-numpy debian package provides a f2py2.7, which seems to be called by the test... if that isn't installed the test will fail, as you can try yourself with a fresh docker debian container:

# start container
docker run --rm -it debian
# inside container:
apt-get update && apt-get install python python-pip python-dev python-virtualenv
virtualenv venv
. venv/bin/activate
pip install numpy nose
python -c 'import numpy ; numpy.test();'

so it's definitely not only a mac issue (will change title), but rather a pip install issue? i think the idea to deduce the f2py name from sys.executable is just not really portable

[charris]

I've got both f2py and f2py2.7, they are just short scripts that import from numpy.

!/usr/bin/python
# See http://cens.ioc.ee/projects/f2py2e/
import os
import sys
for mode in ["g3-numpy", "2e-numeric", "2e-numarray", "2e-numpy"]:
    try:
        i = sys.argv.index("--" + mode)
        del sys.argv[i]
        break
    except ValueError:
        pass
os.environ["NO_SCIPY_IMPORT"] = "f2py"
if mode == "g3-numpy":
    sys.stderr.write("G3 f2py support is not implemented, yet.\\n")
    sys.exit(1)
elif mode == "2e-numeric":
    from f2py2e import main
elif mode == "2e-numarray":
    sys.argv.append("-DNUMARRAY")
    from f2py2e import main
elif mode == "2e-numpy":
    from numpy.f2py import main
else:
    sys.stderr.write("Unknown mode: " + repr(mode) + "\\n")
    sys.exit(1)
main()

The tests should probably skip all that and go directly to numpy.f2py.

[charris]

For instance

python -c'from numpy.f2py import main; main()' -v

seems to do what is intended at the point of failure.

[joernhees]

i think the failing test actually intends to somehow make sure that the f2py script is callable on the command line...

the problem is that it does weird failing magic to determine the different names f2py was installed as... and it assumes that the installing sys.executable name is the same as the one running the tests...

until now in my tests i didn't encounter a single environment setup where no f2py was present but only a f2py2.7...

so maybe to cover the intention of the test better it would be easiest to just trying to call f2py on all *nix systems and rely on the PATH to give us the right script?

[joernhees]

so a simple change of https://github.com/numpy/numpy/blob/master/numpy/tests/test_scripts.py#L67

        f2py_cmd = 'f2py' + basename(sys.executable)[6:]

to

        f2py_cmd = 'f2py'

[pv]

It's not guaranteed that any f2py is available on PATH: pip install --user installs scripts to ~/.local/bin which is not necessarily on path.

[joernhees]

hmm, didn't think about that :-/ but wouldn't the current version with Popen(cmd,...) fail in that case as well?

[pv]

Yes, any test that assumes the f2py script is available on PATH fails in this case. I don't think there's a reliable way to locate where the script got installed, except maybe somehow via pip's install record file.

[matthew-brett]

Chuck - pip is the standard way to install on the Mac, I think.

I'm guessing it's me who put in the test, to test for the problem we had previously where the executable was not correctly installed : #5817

It would be a shame not to test that bug, but of @pv is right, there's no guarantee the install put the script onto the users path. On the other hand, the case of instal --user without the user binary install directory on the path is the only common one I can think of.