Installation fails in current MSYS2

[eonj]

I am using MSYS2 in Windows 10. With MINGW64 shell, Python and pip is available as MSYS2 packages mingw-w64-x86_64-python3 (Python 3.5.2), mingw-w64-x86_64-python3-pip (pip 8.1.2 / manually upgraded to pip 9.0.1) and dependencies installed.

When I try to install numpy via pip, it ends with a failure stack trace ending

    ValueError: python35.dll not found in ['E:/mingw64', 'E:/mingw64/lib', 'C:\\WINDOWS/system32']

saying python35.dll should be found in some location. The DLL filename spec is designated in some places:

numpy/distutils/mingw32ccompiler.py:240:

    dllname = 'python%d%d.dll' % (maj, min)

numpy/distutils/mingw32ccompiler.py:428:

    dll_name = "python%d%d.dll" % tuple(sys.version_info[:2])

In MSYS2, however, Python 3.5 DLL follows libpython<version>m.dll.

$ pacman -Ql mingw-w64-x86_64-python3 | grep dll
mingw-w64-x86_64-python3 /mingw64/bin/libpython3.5m.dll
mingw-w64-x86_64-python3 /mingw64/lib/libpython3.5.dll.a
mingw-w64-x86_64-python3 /mingw64/lib/libpython3.5m.dll.a
mingw-w64-x86_64-python3 /mingw64/lib/python3.5/config-3.5m/libpython3.5.dll.a
mingw-w64-x86_64-python3 /mingw64/lib/python3.5/config-3.5m/libpython3.5m.dll.a
...

The code snippets requiring python<version>.dll is 8~11 years old (known from git blame) and still alive.

Maybe I can write a patch for this, but I don't know any specific DLL path for MinGW, MSYS or Anaconda installation. As I know, previous MinGW/MSYS environment requires Python to be installed separately by the official installer, instead of being installed via pacman. I am slightly afraid that my patch can break any compatibility. Also, I don't know the overall structure of NumPy distutils functionality. What is the appropriate thing I can do? If possible, where is the point I can write a patch on?

[charris]

I believe we gave up on mingw64 and I don't know what the current status is. You might look at the http://mingwpy.github.io/ fork. @matthew-brett Comment?

[charris]

Might also look at #8355 and #8356

[rolk]

@Uni- Please check if the 8362_msys2 branch in my repo works for you. (This of course conflicts with my other pull requests, so there needs to be some more cleanup, even if it does).

[eonj]

@rolk Thanks for your changes, but still looking for unexistent path. It's just because libpython%d.%dm.dll is in /mingw64/bin, not in /mingw64 or /mingw64/lib as the code assumes. It can be easily changed.

A new error is: failure searching /mingw64/libs/python35.def while building /mingw64/libs/libpython35.a(why?).

Here is the new complete error dump:

    openblas_lapack_info:
    Looking for python35.dll
    Building import library (arch=AMD64): "E:/mingw64/libs/libpython35.a" (from E:/mingw64/bin/libpython3.5m.dll)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "E:/tmp/pip-ezmix5eq-build/setup.py", line 391, in <module>
        setup_package()
      File "E:/tmp/pip-ezmix5eq-build/setup.py", line 383, in setup_package
        setup(**metadata)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\core.py", line 135, in setup
        config = configuration()
      File "E:/tmp/pip-ezmix5eq-build/setup.py", line 166, in configuration
        config.add_subpackage('numpy')
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 1002, in add_subpackage
        caller_level = 2)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 971, in get_subpackage
        caller_level = caller_level + 1)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 908, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "numpy/setup.py", line 16, in configuration
        config.add_subpackage('linalg')
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 1002, in add_subpackage
        caller_level = 2)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 971, in get_subpackage
        caller_level = caller_level + 1)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\misc_util.py", line 908, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "numpy/linalg/setup.py", line 26, in configuration
        lapack_info = get_info('lapack_opt', 0)  # and {}
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 380, in get_info
        return cl().get_info(notfound_action)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 572, in get_info
        self.calc_info()
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 1478, in calc_info
        openblas_info = get_info('openblas_lapack')
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 380, in get_info
        return cl().get_info(notfound_action)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 572, in get_info
        self.calc_info()
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 1753, in calc_info
        if not self.check_embedded_lapack(info):
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\system_info.py", line 1769, in check_embedded_lapack
        c = distutils.ccompiler.new_compiler()
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\ccompiler.py", line 596, in new_compiler
        compiler = klass(None, dry_run, force)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\mingw32ccompiler.py", line 93, in __init__
        build_import_library()
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\mingw32ccompiler.py", line 381, in build_import_library
        return _build_import_library_amd64()
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\mingw32ccompiler.py", line 417, in _build_import_library_amd64
        generate_def(dll_file, def_file)
      File "E:/tmp/pip-ezmix5eq-build\numpy\distutils\mingw32ccompiler.py", line 291, in generate_def
        d = open(dfile, 'w')
    FileNotFoundError: [Errno 2] No such file or directory: 'E:/mingw64/libs/python35.def'

Thanks.

[rolk]

@Uni-

A new error is: failure searching /mingw64/libs/python35.def while building /mingw64/libs/libpython35.a(why?).

It now attempts to build the import library, which doesn't go well either. I have added the bin/ directory and rebased it to be on top of the other patches; please try the 8362_msys2 branch again now.

[charris]

@Uni- I've merged #8454, can you check that it now works for You?

[eonj]

Current master doesn't work for me either. I feel so helpless because pasting the error message is the only thing I can do for this project.

Error trace is:

Eon MINGW64 ~/numpy
$ pip3 install .
Processing c:/users/eon/numpy
Installing collected packages: numpy
  Running setup.py install for numpy: started
    Running setup.py install for numpy: finished with status 'error'
    Complete output from command M:/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:/Users/Eon/AppData/Local/Temp/pip-y3gdhqz6-record/install-record.txt --single-version-externally-managed --compile:

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install numpy`   (last NumPy release on PyPi)


    Running from numpy source directory.
    Cythonizing sources
    Processing numpy/random/mtrand/randint_helpers.pxi.in
    Processing numpy/random/mtrand/mtrand.pyx
    blas_opt_info:
    blas_mkl_info:
      libraries mkl_rt not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    blis_info:
      libraries blis not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    openblas_info:
      FOUND:
        libraries = ['openblas', 'openblas']
        library_dirs = ['M:/mingw64/lib']
        language = c
        define_macros = [('HAVE_CBLAS', None)]

      FOUND:
        libraries = ['openblas', 'openblas']
        library_dirs = ['M:/mingw64/lib']
        language = c
        define_macros = [('HAVE_CBLAS', None)]

    non-existing path in 'numpy/distutils': 'site.cfg'
    F2PY Version 2
    lapack_opt_info:
    lapack_mkl_info:
      libraries mkl_rt not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    openblas_lapack_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/setup.py", line 391, in <module>
        setup_package()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/setup.py", line 383, in setup_package
        setup(**metadata)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\core.py", line 135, in setup
        config = configuration()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/setup.py", line 166, in configuration
        config.add_subpackage('numpy')
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 1017, in add_subpackage
        caller_level = 2)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 986, in get_subpackage
        caller_level = caller_level + 1)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 923, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "numpy/setup.py", line 16, in configuration
        config.add_subpackage('linalg')
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 1017, in add_subpackage
        caller_level = 2)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 986, in get_subpackage
        caller_level = caller_level + 1)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\misc_util.py", line 923, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "numpy/linalg/setup.py", line 26, in configuration
        lapack_info = get_info('lapack_opt', 0)  # and {}
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 380, in get_info
        return cl().get_info(notfound_action)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 572, in get_info
        self.calc_info()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 1478, in calc_info
        openblas_info = get_info('openblas_lapack')
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 380, in get_info
        return cl().get_info(notfound_action)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 572, in get_info
        self.calc_info()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 1753, in calc_info
        if not self.check_embedded_lapack(info):
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\system_info.py", line 1769, in check_embedded_lapack
        c = distutils.ccompiler.new_compiler()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\ccompiler.py", line 596, in new_compiler
        compiler = klass(None, dry_run, force)
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\mingw32ccompiler.py", line 107, in __init__
        msvcr_success = build_msvcr_library()
      File "C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build\numpy\distutils\mingw32ccompiler.py", line 372, in build_msvcr_library
        msvcr_dll_name = msvcr_name + '.dll'
    TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

    ----------------------------------------
Command "M:/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:/Users/Eon/AppData/Local/Temp/pip-y3gdhqz6-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:/Users/Eon/AppData/Local/Temp/pip-x53dst2l-build/

I installed cython via pip, MSYS2 pacman packages mingw-w64-x86_64-openblas and mingw-w64-x86_64-lapack.

[eonj]

I think maybe the distutils should not use mingw32ccompiler.py any more for this MinGW-w64 based system (not MinGW based system).

[charris]

I think maybe the distutils should not use mingw32ccompiler.py any more for this MinGW-w64 based system (not MinGW based system).

That makes sense. This isn't my area and I don't use windows so I cannot add much. @rgommers Thoughts?

[rolk]

@Uni-
What does the command M:\mingw64\bin\python3.exe -c "import sys;print (sys.version)" return on your system? To me it seems like the msvc_runtime_version function in misc_utils.py guesses wrong from that string, and it goes downhill from there.

[eonj]

@rolk, 3.5.2 (default, Nov 7 2016, 14:06:02) [GCC 6.2.0 64 bit (AMD64)] in both cmd.exe and MINGW64 shell (MinGW-w64 Win64).

[rolk]

Yup, that was what I thought. The [GCC 6.2.0 part indicates that Python itself is not compiled with the Microsoft compiler, but GCC. Question is whether extensions should now link to libgcc.dll or if it is sufficient to link to msvcrt.dll (I guess not). @Uni- Is there any libgcc.lib files in M:\mingw64 (or subdirectories)?

[eonj]

@rolk, If i guess right, it's here:

$ pacman -Ql mingw-w64-x86_64-gcc | grep libgcc
mingw-w64-x86_64-gcc /mingw64/lib/gcc/x86_64-w64-mingw32/6.3.0/libgcc.a
mingw-w64-x86_64-gcc /mingw64/lib/gcc/x86_64-w64-mingw32/6.3.0/libgcc_eh.a
mingw-w64-x86_64-gcc /mingw64/lib/gcc/x86_64-w64-mingw32/6.3.0/libgcc_s.a

/mingw64 is M:\mingw64 in my case.

[rolk]

/mingw64 is M:\mingw64 in my case.

@Uni- And am I correct in reckoning that M:\mingw64\bin\python3.exe -c "import sys;print (sys.base_prefix)" returns M:\mingw64 ?

[eonj]

@rolk That's right. MSYS2 consists three subsystem; one is MSYS2 (MinGW-w64 built native, uses msys-2,0.dll) having PREFIX /, another is MINGW32 (MinGW-w64 built Win32, uses msvcrt6) having PREFIX /mingw32, and the last is MINGW64 (MinGW-w64 built Win64, uses msvcrt6) having PREFIX /mingw64. / is the installed path.

[rolk]

@Uni- After some thought, here is one thing I want you to test: Instead of trying to find a suitable library, we can assume that if Python is compiled from scratch, then the library/extension will be compiled with the same compiler, and that it will know how to include the right runtime library without further ado. The code is in my 8362b branch, or you can monkey-patch your Numpy with the commit itself.

[eonj]

I have a new error message. @rolk can you have a read?

$ pip3 install .
Processing c:/users/eon/development/numpy
Installing collected packages: numpy
  Running setup.py install for numpy: started
    Running setup.py install for numpy: finished with status 'error'
    Complete output from command M:/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:/Users/Eon/AppData/Local/Temp/pip-exk11nbg-record/install-record.txt --single-version-externally-managed --compile:

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install numpy`   (last NumPy release on PyPi)


    Running from numpy source directory.
    Cythonizing sources
    numpy/random/mtrand/randint_helpers.pxi.in has not changed
    Processing numpy/random/mtrand/mtrand.pyx
    blas_opt_info:
    blas_mkl_info:
      libraries mkl_rt not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    blis_info:
      libraries blis not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    openblas_info:
      FOUND:
        library_dirs = ['M:/mingw64/lib']
        language = c
        libraries = ['openblas', 'openblas']
        define_macros = [('HAVE_CBLAS', None)]

      FOUND:
        library_dirs = ['M:/mingw64/lib']
        language = c
        libraries = ['openblas', 'openblas']
        define_macros = [('HAVE_CBLAS', None)]

    non-existing path in 'numpy/distutils': 'site.cfg'
    F2PY Version 2
    lapack_opt_info:
    lapack_mkl_info:
      libraries mkl_rt not found in ['M:/mingw64/lib', 'C:\\']
      NOT AVAILABLE

    openblas_lapack_info:
    customize Mingw32CCompiler
    C compiler: x86_64-w64-mingw32-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG

    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users
    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon
    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata
    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata/local
    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata/local/temp
    creating C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata/local/temp/tmpp_261oir
    compile options: '-c'
    x86_64-w64-mingw32-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -c C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/source.c -o C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata/local/temp/tmpp_261oir/source.o
    Found executable M:/mingw64/bin/x86_64-w64-mingw32-gcc.exe
    x86_64-w64-mingw32-gcc C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/users/eon/appdata/local/temp/tmpp_261oir/source.o -LM:/mingw64/lib -lopenblas -o C:/Users/Eon/AppData/Local/Temp/tmpp_261oir/a.out.exe
      FOUND:
        library_dirs = ['M:/mingw64/lib']
        language = c
        libraries = ['openblas', 'openblas']
        define_macros = [('HAVE_CBLAS', None)]

      FOUND:
        library_dirs = ['M:/mingw64/lib']
        language = c
        libraries = ['openblas', 'openblas']
        define_macros = [('HAVE_CBLAS', None)]

    M:/mingw64/lib/python3.5\distutils\dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
      warnings.warn(msg)
    running install
    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building py_modules sources
    creating build
    creating build/src.mingw-3.5
    creating build/src.mingw-3.5/numpy
    creating build/src.mingw-3.5/numpy/distutils
    building library "npymath" sources
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build/setup.py", line 391, in <module>
        setup_package()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build/setup.py", line 383, in setup_package
        setup(**metadata)
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\core.py", line 169, in setup
        return old_setup(**new_attr)
      File "M:/mingw64/lib/python3.5\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "M:/mingw64/lib/python3.5\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "M:/mingw64/lib/python3.5\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\install.py", line 62, in run
        r = self.setuptools_run()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\install.py", line 36, in setuptools_run
        return distutils_install.run(self)
      File "M:/mingw64/lib/python3.5\distutils\command\install.py", line 540, in run
        self.run_command('build')
      File "M:/mingw64/lib/python3.5\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "M:/mingw64/lib/python3.5\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\build.py", line 47, in run
        old_build.run(self)
      File "M:/mingw64/lib/python3.5\distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "M:/mingw64/lib/python3.5\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "M:/mingw64/lib/python3.5\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\build_src.py", line 148, in run
        self.build_sources()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\build_src.py", line 159, in build_sources
        self.build_library_sources(*libname_info)
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\build_src.py", line 294, in build_library_sources
        sources = self.generate_sources(sources, (lib_name, build_info))
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\build_src.py", line 377, in generate_sources
        source = func(extension, build_dir)
      File "numpy/core/setup.py", line 663, in get_mathlib_info
        st = config_cmd.try_link('int main(void) { return 0;}')
      File "M:/mingw64/lib/python3.5\distutils\command\config.py", line 243, in try_link
        self._check_compiler()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\command\config.py", line 81, in _check_compiler
        c_compiler=self.compiler)
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\fcompiler\__init__.py", line 837, in new_fcompiler
        load_all_fcompiler_classes()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\fcompiler\__init__.py", line 740, in load_all_fcompiler_classes
        __import__ (module_name)
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\fcompiler\compaq.py", line 57, in <module>
        class CompaqVisualFCompiler(FCompiler):
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\fcompiler\compaq.py", line 81, in CompaqVisualFCompiler
        m.initialize()
      File "C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build\numpy\distutils\msvccompiler.py", line 50, in initialize
        _MSVCCompiler.initialize(self, plat_name)
    TypeError: initialize() takes 1 positional argument but 2 were given

    ----------------------------------------
Command "M:/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:/Users/Eon/AppData/Local/Temp/pip-exk11nbg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:/Users/Eon/AppData/Local/Temp/pip-ek5diz9_-build/

[rolk]

This is like in one of the old Indiana Jones movies, where when the hero has passed one trap, he just finds out that there are more traps!

It seems at first glance that it doesn't find your Fortran compiler, but instead wants to try to use the Intel compiler. Can you run gfortran in this environment? Try to force it to select the GNU compiler by putting these lines in a file named setup.cfg in the same directory as setup.py:

[build]
compiler=mingw32

I am a little confused, though. The signature of the MSVCCompiler.initialize method in the distutils.msvccompiler module has taken no extra arguments since forever: https://github.com/python/cpython/blob/master/Lib/distutils/msvccompiler.py#L248

The MSVCCompiler.initialize method in the distutils.msvc9compiler compiler module however, take a plat_name argument. Could it be that testing always pick this variant? (except when run under MSYS!) Paging @dzagorny for input.

[eonj]

I have gfortran here and it works well. @rolk, with the new setup.cfg, the same error.

It's weird. As you saw, both numpy/distutils/msvccompiler.py and numpy/distutils/msvc9compiler.py use distutils.msvccompiler.MSVCCompiler (imported as _MSVCCompiler) and its method initialize(). with wrong number of arguments! How could it be built for Windows -- even with Cygwin as documented? http://www.scipy.org/scipylib/building/windows.html

Reminder: MSYS2 is a recent fork of Cygwin. + It still tries to use cl instead of gcc.

[rolk]

I reckon it is a copy-paste that was intended to be changed later. I would just merge in b2abbdd from my 8362c branch, and see what happens. If it still insists on using cl, it probably won't end well; I would still attempt to force the compiler in setup.cfg as mentioned above.

[ghost]

@rolk Looks like a bug in numpy/distutils/msvccompiler.py. This code should have the same signature as in python distutils. Your patch b2abbdd should resolve this issue.

[rolk]

@Uni- It may be that the setup picks up a df (disk free) executable from MSYS in another location than /bin, and believes it is the Digital Fortran compiler. If you have such an executable, you can experiment with renaming it temporarily to see if that is a problem here.

[eonj]

@rolk 8362c made this error trace:

(...)\numpy\distutils\mingw32ccompiler.py", line 372, in build_msvcr_library
    msvcr_dll_name = msvcr_name + '.dll'
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

df was in /usr/bin, but changing /usr/bin/df to /usr/bin/df1 did not make any useful changes.

[rolk]

@Uni- This is due to insufficient checking for non-MSVC runtime. This is patched in 48e3e95 in updated branch https://github.com/rolk/numpy/tree/8362c. (Sorry for the slow progress, but I have no way to test what happens next).

[eonj]

Whoa! I think we finally got the build successfully!

$ python3 ./runtests.py
M:/mingw64/lib/python3.5\importlib\_bootstrap.py:222: Warning: Numpy built with MINGW-W64 on Windows 64 bits is experimental, and only available for
testing. You are advised not to use it for production.

CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS
  return f(*args, **kwds)

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Building, see build.log...
Build OK
Running unit tests for numpy
NumPy version 1.13.0.dev0+48e3e95
NumPy relaxed strides checking option: True
NumPy is installed in C:/Users/Eon/Development/numpy/build/testenv/lib/python3.5/site-packages\numpy
Python version 3.5.2 (default, Nov  7 2016, 14:06:02)  [GCC 6.2.0 64 bit (AMD64)]
nose version 1.3.7

The slow progress was all my fault; I should have tested your patches more uprightly.

Anyway, it was successful and thank you very much!

[eonj]

$ python3
Python 3.5.2 (default, Nov  7 2016, 14:06:02)  [GCC 6.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'numpy' is not defined
>>> import numpy
M:/mingw64/lib/python3.5\importlib\_bootstrap.py:222: Warning: Numpy built with MINGW-W64 on Windows 64 bits is experimental, and only available for
testing. You are advised not to use it for production.

CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS
  return f(*args, **kwds)
>>> x = numpy.arange(0.0, 2*numpy.pi, 0.1)
>>> x
array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ,
        1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,  2.1,
        2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,  3.2,
        3.3,  3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,  4.3,
        4.4,  4.5,  4.6,  4.7,  4.8,  4.9,  5. ,  5.1,  5.2,  5.3,  5.4,
        5.5,  5.6,  5.7,  5.8,  5.9,  6. ,  6.1,  6.2])

[rolk]

@Uni- Looks good then? I will wrap up all the changes we did and post them as patch requests. Thank you for your persistence; too many people give up on first hurdle, which is why progress with MSYS is so slow I guess.

[eonj]

It looks good and works fine as I test some matrix multi and inv codes 👍

[rolk]

I have created pull request #8675 that contains the remaining fixes from this issue. When this is merged, you should be able to build the master branch without any further patches.

@Uni- In summary: Does this mean that Numpy can now be installed in MSYS2 with simply pacman mingw-w64-x86_64-{openblas,lapack,python3,python3-pip} first?

[rgommers]

Looks like it's all resolved with gh-8675, thanks all!

[GlenNicholls]

What is the workaround to install numpy on MSYS2? I installed the packages outlined by @rolk , but I still see these errors:

Glen.Nicholls@CL1073 MINGW64 /d/dev/test_dev/dsss_branch/test_utils
$ pip install numpy --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host pypi.org
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/b6/d6/be8f975f5322336f62371c9abeb936d592c98c047ad63035f1b38ae08efe/numpy-1.17.3.zip (6.4MB)
     |████████████████████████████████| 6.4MB 20kB/s
Installing collected packages: numpy
    Running setup.py install for numpy ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.exe -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8c70yr1f/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8c70yr1f/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sd9x7t4g/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-8c70yr1f/numpy/
    Complete output (251 lines):
    Running from numpy source directory.

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install numpy`   (last NumPy release on PyPi)


    blas_opt_info:
    blas_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    blis_info:
    customize UnixCCompiler
      libraries blis not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_3_10_blas_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries tatlas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE
    
    atlas_3_10_blas_info:
    customize UnixCCompiler
      libraries satlas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_blas_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_blas_info:
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE
    
    accelerate_info:
      NOT AVAILABLE

    /tmp/pip-install-8c70yr1f/numpy/numpy/distutils/system_info.py:690: UserWarning:
        Optimized (vendor) Blas libraries are not found.
        Falls back to netlib Blas library which has worse performance.
        A better performance should be easily gained by switching
        Blas library.
      self.calc_info()
    blas_info:
    customize UnixCCompiler
      libraries blas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    /tmp/pip-install-8c70yr1f/numpy/numpy/distutils/system_info.py:690: UserWarning:
        Blas (http://www.netlib.org/blas/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [blas]) or by setting
        the BLAS environment variable.
      self.calc_info()
    blas_src_info:
      NOT AVAILABLE

    /tmp/pip-install-8c70yr1f/numpy/numpy/distutils/system_info.py:690: UserWarning:
        Blas (http://www.netlib.org/blas/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [blas_src]) or by setting
        the BLAS_SRC environment variable.
      self.calc_info()
      NOT AVAILABLE

    non-existing path in 'numpy/distutils': 'site.cfg'
    lapack_opt_info:
    lapack_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_lapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_clapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas,lapack not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    flame_info:
    customize UnixCCompiler
      libraries flame not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_3_10_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      NOT AVAILABLE

    atlas_3_10_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_3_10_info'>
      NOT AVAILABLE

    atlas_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_threads_info'>
      NOT AVAILABLE

    atlas_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_info'>
      NOT AVAILABLE
    
    lapack_info:
    customize UnixCCompiler
      libraries lapack not found in ['/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    /tmp/pip-install-8c70yr1f/numpy/numpy/distutils/system_info.py:1712: UserWarning:
        Lapack (http://www.netlib.org/lapack/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [lapack]) or by setting
        the LAPACK environment variable.
      if getattr(self, '_calc_info_{}'.format(lapack))():
    lapack_src_info:
      NOT AVAILABLE

    /tmp/pip-install-8c70yr1f/numpy/numpy/distutils/system_info.py:1712: UserWarning:
        Lapack (http://www.netlib.org/lapack/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [lapack_src]) or by setting
        the LAPACK_SRC environment variable.
      if getattr(self, '_calc_info_{}'.format(lapack))():
      NOT AVAILABLE

    /usr/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
      warnings.warn(msg)
    running install
    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building py_modules sources
    creating build
    creating build/src.mingw64_nt-10.0-18362-3.0.7-338.x86_64-x86_64-3.7
    creating build/src.mingw64_nt-10.0-18362-3.0.7-338.x86_64-x86_64-3.7/numpy
    creating build/src.mingw64_nt-10.0-18362-3.0.7-338.x86_64-x86_64-3.7/numpy/distutils
    building library "npymath" sources
    get_default_fcompiler: matching types: '['gnu', 'gnu95']'
    customize GnuFCompiler
    Could not locate executable g77
    Could not locate executable f77
    customize Gnu95FCompiler
    Found executable /mingw64/bin/gfortran
    customize Gnu95FCompiler
    customize Gnu95FCompiler using config
    C compiler: x86_64-pc-msys-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.7m -c'
    x86_64-pc-msys-gcc: _configtest.c
    failure.
    removing: _configtest.c _configtest.o
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-8c70yr1f/numpy/setup.py", line 443, in <module>
        setup_package()
      File "/tmp/pip-install-8c70yr1f/numpy/setup.py", line 435, in setup_package
        setup(**metadata)
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/core.py", line 171, in setup
        return old_setup(**new_attr)
      File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/install.py", line 62, in run
        r = self.setuptools_run()
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
        return distutils_install.run(self)
      File "/usr/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/build.py", line 47, in run
        old_build.run(self)
      File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/build_src.py", line 142, in run
        self.build_sources()
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/build_src.py", line 153, in build_sources
        self.build_library_sources(*libname_info)
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/build_src.py", line 286, in build_library_sources
        sources = self.generate_sources(sources, (lib_name, build_info))
      File "/tmp/pip-install-8c70yr1f/numpy/numpy/distutils/command/build_src.py", line 369, in generate_sources
        source = func(extension, build_dir)
      File "numpy/core/setup.py", line 669, in get_mathlib_info
        raise RuntimeError("Broken toolchain: cannot link a simple C program")
    RuntimeError: Broken toolchain: cannot link a simple C program
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.exe -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8c70yr1f/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8c70yr1f/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sd9x7t4g/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

[mattip]

This issue is closed. If you want to pursue building numpy with mingw64 (probably a difficult task), please open a new one. Describe exactly what you installed, and the error message you got. You may have some luck by setting CC to your compiler path.