error during fortran version detection

[akamaus]

During installation of DIRECT optimizer I stumbled upon an error apparently related to numpy's distutils:

% sudo pip3 install  direct
Installing collected packages: direct
  Running setup.py install for direct ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-_dyg83k3/direct/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-yvn89jic/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building extension "DIRECT.direct" sources
    f2py options: []
      adding 'build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.c' to sources.
      adding 'build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src' to include_dirs.
      adding 'build/src.linux-x86_64-3.5/src/direct-f2pywrappers.f' to sources.
    build_src: building npy-pkg config files
    running build_py
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/DIRECT
    copying DIRECT/__init__.py -> build/lib.linux-x86_64-3.5/DIRECT
    running build_ext
    customize UnixCCompiler
    customize UnixCCompiler using build_ext
    get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
    customize Gnu95FCompiler
    Found executable /usr/bin/gfortran
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_dyg83k3/direct/setup.py", line 54, in <module>
        classifiers =classifiers)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/core.py", line 169, in setup
        return old_setup(**new_attr)
      File "/home/vyal/.local/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/command/install.py", line 62, in run
        r = self.setuptools_run()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/command/install.py", line 36, in setuptools_run
        return distutils_install.run(self)
      File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
        self.run_command('build')
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/command/build.py", line 47, in run
        old_build.run(self)
      File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/command/build_ext.py", line 223, in run
        c_compiler=self.compiler)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 873, in new_fcompiler
        c_compiler=c_compiler)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 847, in get_default_fcompiler
        c_compiler=c_compiler)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 794, in _find_existing_fcompiler
        c.customize(dist)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 499, in customize
        get_flags('opt', oflags)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 490, in get_flags
        flags.extend(getattr(self.flag_vars, tag))
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/environment.py", line 39, in __getattr__
        return self._get_var(name, conf_desc)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/environment.py", line 53, in _get_var
        var = self._hook_handler(name, hook)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 697, in _environment_hook
        return hook()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/gnu.py", line 233, in get_flags_opt
        v = self.get_version()
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/__init__.py", line 428, in get_version
        version = CCompiler.get_version(self, force=force, ok_status=ok_status)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/ccompiler.py", line 89, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/ccompiler.py", line 627, in CCompiler_get_version
        version = matcher(output)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/gnu.py", line 270, in version_match
        v = self.gnu_version_match(version_string)
      File "/usr/local/lib/python3.5/dist-packages/numpy/distutils/fcompiler/gnu.py", line 83, in gnu_version_match
        raise ValueError(err + version_string)
    ValueError: A valid Fortran version was not found in this string:
    OK
    4.9.3

I tried gfortran5 with the similar result.

Version information:

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.15.1 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

$ lsb_release -a                                                                                            
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

[akamaus]

I found the immediate culprit.

during compiler detection _exec_command (at numpy/distutils/exec_command.py:223)
is called with command = ['/bin/zsh', '-c', '/usr/bin/gfortran -dumpversion']

For some reason zsh adds 'OK' if called using Popen, this breaks version detection logic and other things (subsequent compilation failed then I tried dirty-patching).

import subprocess
command =  ['/bin/zsh', '-c', '/usr/bin/gfortran -dumpversion']
pp = subprocess.Popen(command, shell=False, env=None, stdout=subprocess.PIPE)
pp.communicate()
(b'OK\n5.4.0\n', None)

Behavior is different if the same command is run from shell:

$ zsh -c '/usr/bin/gfortran -dumpversion'
5.4.0

In my case unset SHELL was enough to make DIRECT install successfully.

[charris]

I'll wait until you are satisfied that you have found the problem. Note that we search the version strings, so it is certainly possible to fix up that part of them.

[akamaus]

Yeah, of course at first I tried to fix version search. Other weird problems appeared during subsequent compilation, like this:

/usr/bin/gfortran -Wall -g -Wall -g -shared build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/directmodule.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.o build/temp.linux-x86_64-3.5/src/DIRect.o build/temp.linux-x86_64-3.5/src/DIRserial.o build/temp.linux-x86_64-3.5/src/DIRsubrout.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/direct-f2pywrappers.o -LOK
/usr/lib/gcc/x86_64-linux-gnu/5 -LOK
/usr/lib/gcc/x86_64-linux-gnu/5 -lgfortran -o build/lib.linux-x86_64-3.5/DIRECT/direct.cpython-35m-x86_64-linux-gnu.so
OK
zsh:2: permission denied: /usr/lib/gcc/x86_64-linux-gnu/5
zsh:3: permission denied: /usr/lib/gcc/x86_64-linux-gnu/5
OK
zsh:2: permission denied: /usr/lib/gcc/x86_64-linux-gnu/5
zsh:3: permission denied: /usr/lib/gcc/x86_64-linux-gnu/5
error: Command "/usr/bin/gfortran -Wall -g -Wall -g -shared build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/directmodule.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.o build/temp.linux-x86_64-3.5/src/DIRect.o build/temp.linux-x86_64-3.5/src/DIRserial.o build/temp.linux-x86_64-3.5/src/DIRsubrout.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/direct-f2pywrappers.o -LOK
/usr/lib/gcc/x86_64-linux-gnu/5 -LOK
/usr/lib/gcc/x86_64-linux-gnu/5 -lgfortran -o build/lib.linux-x86_64-3.5/DIRECT/direct.cpython-35m-x86_64-linux-gnu.so" failed with exit status 126

I guess the most conservative fix is just to use /bin/sh if it's available. Or be prepared for surprises zsh,ksh and others may prepare.

[QuLogic]

Why does it even need the shell wrapper? subprocess.Popen(['gcc', '-dumpversion'], ...) should be fine, no?

[HaoZeke]

This should be fixed; cannot reproduce.