Failure to install h5py in MSYS2 #898

ahmwb · 2017-06-19T15:00:54Z

Using MSYS2 from http://msys2.github.io/, steps to reproduce:

pacman -S mingw-w64-x86_64-python3-pip
python3 -m pip install numpy
python3 -m pip install h5py

Messages:

C:/msys64/mingw64/lib/python3.5\importlib_bootstrap.py:222: Warning: Numpy built with MINGW-W64 on Windows 64 bits is experimental, and only available for
testing. You are advised not to use it for production.

CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS
  return f(*args, **kwds)
Autodetection skipped [[WinError 126] The specified module could not be found]
********************************************************************************
                       Summary of the h5py configuration

    Path to HDF5: None
    HDF5 Version: '1.8.4'
     MPI Enabled: False
Rebuild Required: True

Errors:

**building 'h5py.defs' extension
creating build/temp.mingw-3.5
creating build/temp.mingw-3.5/Users
creating build/temp.mingw-3.5/Users/ahm
creating build/temp.mingw-3.5/Users/ahm/AppData
creating build/temp.mingw-3.5/Users/ahm/AppData/Local
creating build/temp.mingw-3.5/Users/ahm/AppData/Local/Temp
creating build/temp.mingw-3.5/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee
creating build/temp.mingw-3.5/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py
creating build/temp.mingw-3.5/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py
C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -DH5_USE_ 16_API -D_HDF5USEDLL_ -DH5_BUILT_AS_DYNAMIC_LIB -IC:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/lzf -IC:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/windows -IC:/msys64/ mingw64/lib/python3.5/site-packages\numpy\core/include -IC:/msys64/mingw64/include/python3.5m -c C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c -o build/temp.mingw-3.5 /Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.o
In file included from C:/msys64/mingw64/x86_64-w64-mingw32/include/inttypes.h:12:0,
from C:/msys64/mingw64/include/python3.5m/pyport.h:9,
from C:/msys64/mingw64/include/python3.5m/Python.h:50,
from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c:41:
C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/msys64/mingw64/include/python3.5m/pyport.h:13:0,
from C:/msys64/mingw64/include/python3.5m/Python.h:50,
from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c:41:
C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c:472:0:
C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/ndarraytypes.h:1809:0,
from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/ndarrayobject.h:18,
from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/arrayobject.h:4,
from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/api_compat.h:26,
from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c:473:
C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/npy_1_7_deprecated_api.h:12:9: note: #pragma message: C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/in clude/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#pragma message(WARN___LOC_"Using deprecated NumPy API, disable it by "
^~~~~~~
In file included from C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/defs.c:473:0:
C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
error: command 'C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe' failed with exit status 1

----------------------------------------

Command "C:/msys64/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;file='C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/setup.py';f=getattr(tokenize, 'open', open)( file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:/Users/ahm/AppData/Local/Temp/pip-v_jipatd-record/install-record.txt --sin gle-version-externally-managed --compile" failed with error code 1 in C:/Users/ahm/AppData/Local/Temp/pip-build-3x1v0qee/h5py/**

The text was updated successfully, but these errors were encountered:

aragilar · 2017-06-20T09:54:48Z

It looks like multiple things are going on here, first that HDF5 is not installed (or detected), and that there is an issue about which headers are to be used (what's the state of stdint.h with msys2?).

ahmwb · 2017-06-20T10:57:23Z

Well the h5py package claims it has detected HDF5 Version: '1.8.4' in its pip build. It must be something wrong about it setting the path to the detected version (1.8.4 is quite old, but ok).

stdint.h is just fine under msys2, it compiles without errors:

cat test.c
#include <stdint.h>

int main() { return 0; }
~/tmp/X $ gcc -c test.c
~/tmp/X $

So no problem on that part. However, it seems h5py comes with a header file that compiles only with the Visual Studio compiler, not with the mingw compiler that comes with msys.

aragilar · 2017-06-20T13:43:29Z

Sorry, I should have been more explicit, does stdint.h in msys2 match what HDF5 expects on Windows (and what about numpy/cython)?

The version '1.8.4' is the lowest version we support, and the lack of path to me implies HDF5 wasn't found (you need HDF5_DIR on Windows).

ahmwb · 2017-06-20T15:21:18Z

Ok, I did not know setting HDF5_DIR was required (please update the message for this), but now I set it to HDF5 1.8.19 , the path is picked up, but it still reports 1.8.4 and while the HDF5-related error message is gone, it still finds the wrong header file that comes with h5py for Visual Studio:

c:/msys64/tmp/pip-build-3qgfut_5/h5py/.eggs/cython-0.25.2-py3.5-mingw.egg\Cython\Distutils\old_build_ext.py:40: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
  "Cython.Distutils.old_build_ext does not properly handle dependencies "
C:/msys64/mingw64/lib/python3.5\importlib\_bootstrap.py:222: Warning: Numpy built with MINGW-W64 on Windows 64 bits is experimental, and only available for
testing. You are advised not to use it for production.

CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS
  return f(*args, **kwds)
Autodetection skipped [[WinError 126] The specified module could not be found]
********************************************************************************
                       Summary of the h5py configuration

    Path to HDF5: 'C:/msys64/home/ahm/sw'
    HDF5 Version: '1.8.4'
     MPI Enabled: False
Rebuild Required: True

********************************************************************************
Executing api_gen rebuild of defs
Executing cythonize()

....

[22/22] Cythonizing C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/utils.pyx
building 'h5py.defs' extension
creating build/temp.mingw-3.5
creating build/temp.mingw-3.5/msys64
creating build/temp.mingw-3.5/msys64/tmp
creating build/temp.mingw-3.5/msys64/tmp/pip-build-3qgfut_5
creating build/temp.mingw-3.5/msys64/tmp/pip-build-3qgfut_5/h5py
creating build/temp.mingw-3.5/msys64/tmp/pip-build-3qgfut_5/h5py/h5py
C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -DH5_USE_16_API -D_HDF5USEDLL_ -DH5_BUILT_AS_DYNAMIC_LIB -IC:/msys64/home/ahm/sw/include -IC:/msys64/tmp/pip-build-3qgfut_5/h5py/lzf -IC:/msys64/tmp/pip-build-3qgfut_5/h5py/windows -IC:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include -IC:/msys64/mingw64/include/python3.5m -c C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c -o build/temp.mingw-3.5/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.o
In file included from C:/msys64/mingw64/x86_64-w64-mingw32/include/inttypes.h:12:0,
from C:/msys64/mingw64/include/python3.5m/pyport.h:9,
from C:/msys64/mingw64/include/python3.5m/Python.h:50,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c:48:
C:/msys64/tmp/pip-build-3qgfut_5/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/msys64/mingw64/include/python3.5m/pyport.h:13:0,
from C:/msys64/mingw64/include/python3.5m/Python.h:50,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c:48:
C:/msys64/tmp/pip-build-3qgfut_5/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c:479:0:
C:/msys64/tmp/pip-build-3qgfut_5/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
In file included from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/ndarraytypes.h:1809:0,
from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/ndarrayobject.h:18,
from C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/arrayobject.h:4,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/api_compat.h:26,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c:480:
C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/npy_1_7_deprecated_api.h:12:9: note: #pragma message: C:/msys64/mingw64/lib/python3.5/site-packages\numpy\core/include/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#pragma message(WARN___LOC_"Using deprecated NumPy API, disable it by "
^~~~~~~
In file included from C:/msys64/home/ahm/sw/include/H5public.h:45:0,
from C:/msys64/home/ahm/sw/include/hdf5.h:22,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/api_compat.h:27,
from C:/msys64/tmp/pip-build-3qgfut_5/h5py/h5py/defs.c:480:
C:/msys64/tmp/pip-build-3qgfut_5/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"
#error "Use this header only with Microsoft Visual C++ compilers!"
^~~~~
error: command 'C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe' failed with exit status 1

----------------------------------------

Command "C:/msys64/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;file='C:/msys64/tmp/pip-build-3qgfut_5/h5py/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:/msys64/tmp/pip-47yqqcmt-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:/msys64/tmp/pip-build-3qgfut_5/h5py/

aragilar · 2017-06-21T01:44:42Z

You're going to need to patch https://github.com/h5py/h5py/blob/master/setup_build.py#L45 to deal with what paths to include. Also, Windows lacks auto-detection support (as it's far simpler to rely on passing the correct config, see http://docs.h5py.org/en/latest/build.html#custom-installation), so you need to pass in HDF5_VERSION also.

ahmwb · 2017-06-26T13:25:59Z

Does that mean I need to git checkout h5py and modify it?

The main issue is that I am trying to do

python3 -m pip install h5py

as it is provided by the MSYS2 package manager, which evidently leads to problems while it should work out of the box. Installing numpy worked fine as wll, can't h5py be provided as binary for those platforms as well? Particularly when installing h5py on multiple machines it's quite a hassle to do all such manual work. It looks to me the problem is with h5py itself providing a header file and path to a header file that is not suitable for the compiler:

C:/msys64/tmp/pip-build-3qgfut_5/h5py/windows/stdint.h:33:2: error: #error "Use this header only with Microsoft Visual C++ compilers!"

By setting the environment variables appropriatly, I can get it to find the correct HDF5 version:

                       Summary of the h5py configuration

    Path to HDF5: 'C:/msys64/home/ahm/sw'
    HDF5 Version: '1.8.19'

So that part is fine. But how can it be made to not use this h5py-provided header file when installing via pip ?

aragilar · 2017-06-26T14:03:20Z

You're going to need to checkout and modify yes (see my previous comment as to where). This is because there isn't MSYS2 support in h5py. We do provide wheels for Windows (see https://pypi.python.org/pypi/h5py/2.7.0), but I have no idea whether they support MSYS2.

What would be needed to support MSYS2 in h5py is:

a patch which sets up the correct headers for MSYS2 without breaking the existing Windows support
a patch which tests building with MSYS2 on appveyor (or other CI)

and for wheels we need:

PyPI support for MSYS2 wheels
clarification about what needs to be included in the wheel (do we include HDF5 or not, how stable are the ABI we're dealing with, etc.)

I guess you know this, but MSYS2 is not a supported platform for CPython (Visual Studio is explicitly the only supported platform), and distutils support for it is patchy (see https://bugs.python.org/issue4709 and links within for some background), so out-of-the-box support is going to need someone who uses MSYS2 to produce a patch (as happened with numpy in numpy/numpy#8362).

takluyver · 2018-10-16T13:27:21Z

I'd suggest bumping this to a future milestone, or removing the milestone entirely.

takluyver · 2019-04-26T10:01:42Z

I'm going to remove the 2.10 milestone from this, because no-one is working on it. This shouldn't discourage anyone who wants to work on it, but we can't realistically include it in release planning.

tacaswell added this to the 2.8 milestone Aug 20, 2017

aragilar added the pull-request-welcome label Aug 30, 2017

tacaswell modified the milestones: 2.8, 2.9 Feb 11, 2018

tacaswell modified the milestones: 2.9, 2.10 Oct 16, 2018

takluyver removed this from the 2.10 milestone Apr 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure to install h5py in MSYS2 #898

Failure to install h5py in MSYS2 #898

ahmwb commented Jun 19, 2017

aragilar commented Jun 20, 2017

ahmwb commented Jun 20, 2017

aragilar commented Jun 20, 2017

ahmwb commented Jun 20, 2017 •

edited

aragilar commented Jun 21, 2017

ahmwb commented Jun 26, 2017

aragilar commented Jun 26, 2017

takluyver commented Oct 16, 2018

takluyver commented Apr 26, 2019

Failure to install h5py in MSYS2 #898

Failure to install h5py in MSYS2 #898

Comments

ahmwb commented Jun 19, 2017

aragilar commented Jun 20, 2017

ahmwb commented Jun 20, 2017

aragilar commented Jun 20, 2017

ahmwb commented Jun 20, 2017 • edited

aragilar commented Jun 21, 2017

ahmwb commented Jun 26, 2017

aragilar commented Jun 26, 2017

takluyver commented Oct 16, 2018

takluyver commented Apr 26, 2019

ahmwb commented Jun 20, 2017 •

edited