Cannot compile on MacOS on Apple Silicon (M2 Max)

[nucflash]

Hi, I'm installing ratarmount which has indexed_zstd as dependency. It used to build and install fine on my Intel-based MacBook Pro, however, on my M2 Max (Apple Silicon) MacBook Pro it fails with this error:

Building wheel for indexed-zstd (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for indexed-zstd (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      running bdist_wheel
      running build
      running build_py
      file indexed_zstd.py (for module indexed_zstd) not found
      file indexed_zstd.py (for module indexed_zstd) not found
      running build_clib
      building 'zstd_zeek' library
      creating build
      creating build/temp.macosx-13.4-arm64-cpython-310
      creating build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd
      creating build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd/libzstd-seek
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/redacted/usr/include -I/redacted/usr/include -I. -c indexed_zstd/libzstd-seek/zstd-seek.c -o build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd/libzstd-seek/zstd-seek.o
      In file included from indexed_zstd/libzstd-seek/zstd-seek.c:18:
      indexed_zstd/libzstd-seek/zstd-seek.h:20:10: fatal error: 'zstd.h' file not found
      #include <zstd.h>
               ^~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

Any advice is welcome. Thanks for your great work!

[mxmlnkn]

It looks like the wheels for your system are missing on PyPI. I see wheels for MacOS X 10.9 and 11.0. Which version do you use? As a workaround, you can install the Zstandard development package in order to install indexed_zstd from source. brew install zstd should be sufficient to fix your posted error about the missing zstd.h file. In the future, it would be better though if wheels for your particular MacOS version are uploaded to PyPI.

[martinellimarco]

I second what @mxmlnkn said.
It seems that github actions doesn't support Apple Silicon yet.
It should be possible to compile the ARM wheels on an Intel macOS runner setting CIBW_ARCHS_MACOS="x86_64 universal2 arm64" but it's not possible to test the result according to the docs.
I can't test it either as I don't have an Apple Silicon machine.

I will try it anyway and see if it build at least.

[martinellimarco]

I made a branch to test this and it seems to build fine with CIBW_ARCHS_MACOS: "x86_64 arm64".
The action does use brew install zstd to install the dependencies so I'm positive you'll be able to install it this way.

[nucflash]

Hi mxmlnkn, martinellimarco, thanks for the quick replies.

I'm using macOS 13.4. zstd seems to be already installed by brew:

❯ brew install zstd
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/command-not-found and homebrew/cask).

Warning: zstd 1.5.5 is already installed and up-to-date.
To reinstall 1.5.5, run:
  brew reinstall zstd

But I re-installed it:

❯ brew reinstall zstd
==> Fetching zstd
==> Downloading https://redacted/brew-bottles-core-binaries-local/homebrew-core/zstd/289b40f5/zstd-1.5.5.arm64_ventura.bottle.1.tar.gz
Already downloaded: /Users/manos/Library/Caches/Homebrew/downloads/56ef35482d6395fe8ccf4457fd35ac849fee0288ab2d2d1c334b79d332091920--zstd-1.5.5.arm64_ventura.bottle.1.tar.gz
==> Reinstalling zstd 
==> Pouring zstd-1.5.5.arm64_ventura.bottle.1.tar.gz
🍺  /opt/homebrew/Cellar/zstd/1.5.5: 31 files, 2.3MB
==> Running `brew cleanup zstd`...

Even then, the error persists:

❯ pip install indexed-zstd
Looking in indexes: https://pypi.org/simple
Collecting indexed-zstd
  Using cached https://pypi.org/packages/packages/90/38/fb4db668c97832a4532ff810e630306b59b30a487d7169ee3c28d7ad6a78/indexed_zstd-1.6.0.tar.gz (66 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: indexed-zstd
  Building wheel for indexed-zstd (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for indexed-zstd (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      running bdist_wheel
      running build
      running build_py
      file indexed_zstd.py (for module indexed_zstd) not found
      file indexed_zstd.py (for module indexed_zstd) not found
      running build_clib
      building 'zstd_zeek' library
      creating build
      creating build/temp.macosx-13.4-arm64-cpython-310
      creating build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd
      creating build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd/libzstd-seek
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/redacted/usr/include -I/redacted/usr/include -I. -c indexed_zstd/libzstd-seek/zstd-seek.c -o build/temp.macosx-13.4-arm64-cpython-310/indexed_zstd/libzstd-seek/zstd-seek.o
      In file included from indexed_zstd/libzstd-seek/zstd-seek.c:18:
      indexed_zstd/libzstd-seek/zstd-seek.h:20:10: fatal error: 'zstd.h' file not found
      #include <zstd.h>
               ^~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for indexed-zstd
Failed to build indexed-zstd
ERROR: Could not build wheels for indexed-zstd, which is required to install pyproject.toml-based projects

I'm on python==3.10.11 using pyenv, and pip==23.0.1

[nucflash]

I had to enable Permissive Security Policy on my Apple Silicon Mac so that the FUSE kernel extension was allowed to run. Afterwards compilation ran successfully.