Fails to build from source on MacOS 11 + M1

[neilconway]

• Operating System: MacOS 11.2
• Python Version: 3.9.1
• h5py version: 3.1.0
• HDF5 version: 1.12.0

Attempting to install h5py via pip fails for ARM Macs; there is no prebuilt wheel for this platform, so pip attempts to build h5py from source. However, this fails because it attempts to use Numpy 1.19.3 (when on Python 3.9), but Numpy 1.19.3 does not build on M1 Macs out of the box (build log: h5py_build_fail.log).

If you modify setup.py so that h5py builds against Numpy 1.20.1 instead (which works on M1 Macs), h5py builds successfully.

Dumb example patch that fixes the build but isn't intended to be actually merged:

diff --git a/setup.py b/setup.py
index cc583bc9..9c0400cc 100755
--- a/setup.py
+++ b/setup.py
@@ -29,7 +29,7 @@ NUMPY_MIN_VERSIONS = [
     # Numpy    Python
     ('1.14.5', "=='3.7'"),
     ('1.17.5', "=='3.8'"),
-    ('1.19.3', ">='3.9'"),
+    ('1.20.1', ">='3.9'"),
 ]

 # these are required to use h5py

[takluyver]

What's the value of platform.machine() on an ARM Mac? And is sys.platform still 'darwin'?

[neilconway]

@takluyver

$ python
Python 3.9.1 (default, Jan 21 2021, 10:35:36)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.platform
'darwin'
>>> import platform
>>> platform.machine()
'arm64'

[nathanblair]

That's an interesting result. Maybe upstream Python needs an enumeration added for darwin-arm64 for the platform.machine() method?

[neilconway]

Maybe upstream Python needs an enumeration added for darwin-arm64 for the platform.machine() method?

Sorry, I don't follow what you're suggesting -- can you clarify? At a glance, the current results from the platform package seem about what I'd expect.

[takluyver]

I think to fix this we'll need more specific rules for when to use what version of numpy, like in oldest_supported_numpy or in scipy.

In the meantime, you can work around it by following the steps for development installation to build with an already-installed numpy rather than creating a temporary build environment.

[nathanblair]

Maybe upstream Python needs an enumeration added for darwin-arm64 for the platform.machine() method?

Sorry, I don't follow what you're suggesting -- can you clarify? At a glance, the current results from the platform package seem about what I'd expect.

I was mistaken and didn't see that platform did in fact already resolve with darwin. I thought machine() was expected to respond with the kernel type and not the arch 😅

So please disregard!

[WillTrojak]

Sorry to reopen an old issue, but I had this same problem trying to install h5py and I think I came up with a bit of dirty fix. More than happy for someone to point out a more elegant fix.

The issue for me at least seemed to be with numpy 1.19.3 failing to build. With homebrew numpy 1.20.3 is available, but even with that install the build for h5py will try and build 1.19.3.

By cloning the current master, then bumping the numpy version to 1.20.3 in setup.py, this seemed to get round the issue.

This may have some side effects, but at least it works for now.

[jiegec]

I found an alternative solution:

HDF5_DIR=/opt/homebrew/opt/hdf5 pip3 install --no-build-isolation h5py

[0xrubes]

@WillTrojak Your fix worked! I also needed to do export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1/ and then follow the steps from here to build it

[takluyver]

Just to note, the HDF5_DIR environment variable is documented here ('To specify where to find HDF5'):

https://docs.h5py.org/en/stable/build.html#custom-installation

[takluyver]

This should be fixed now, in that it should get a supported version of NumPy when it tries to build. We don't have pre-built Mac ARM wheels yet, so it will still need to build from source.

[jiahaom]

https://stackoverflow.com/a/70588044/15266409

[rainwoodman]

My understanding is that without the prebuilt arm wheels, pip install h5py needs an active hdf5 installation on the system, but with the prebuilt arm wheels, there is no need for an active hdf5 installation on the system. Is this claim true?

Do we have a plan to upload pre-built arm wheels?

[craigvanden]

I have found that pip install only works if you have installed hdf5 separately first on the Mac M1.
This was possible through brew but not pip.

Steps:

brew install hdf5
export HDF5_DIR=<path/to/homebrew/Cellar/hdf5/version>
pip install h5py

[takluyver]

@rainwoodman that's right, yes. The wheels include a copy of HDF5. @matham just added wheels for ARM Macs in #2065 (and a couple of follow up PRs), so we'll hopefully upload these with the next release.

[rainwoodman]

Thanks for the prompt update!

[haiyan-yhy]

I found an alternative solution:

HDF5_DIR=/opt/homebrew/opt/hdf5 pip3 install --no-build-isolation h5py

good