Making cffi work on Mac M1

[almarklein]

See (the top post of) #136 for the bigger picture. I'm trying to collect what we know here (and may update).

It looks like we have two problems with CFFI.

1. It won't install correctly by default

Problem

There are no binary wheels provided for macos_arm64, so it tries to use the x86 one (or something like this). Also see the corresponding issue: https://foss.heptapod.net/pypy/cffi/-/issues/506 - a workaround has been proposed, and it looks like they'll wait for someone else to fix it.

This error that you'd get:

E ImportError: dlopen(/Users/xx/pypoetry/virtualenvs/wgpu-5bSf_T1V-py3.9/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
E Referenced from: /Users/xx/pypoetry/virtualenvs/wgpu-5bSf_T1V-py3.9/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
E Expected in: flat namespace
E in /Users/xx/pypoetry/virtualenvs/wgpu-5bSf_T1V-py3.9/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so

Solutions

Ideally someone (not me) steps up and helps cffi build wheels for macos_arm64. No idea how hard this is.

A workaround has been proposed:

pip install cffi --no-binary :all:

And a somewhat more complicated version:

LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:

I've heard mixed results so far though.

It looks like installing via conda produces a working version.

If we can confirm that the above workaround works, we can add that in our installation instructions: "bla use conda or pip install cffi --no-binary :all:".

2. Callbacks are broken

Something to do with how callbacks work in cffi, and Mac nagging that the binary is unsigned. Read more here: https://cffi.readthedocs.io/en/latest/using.html#callbacks-old-style

The error that you'd get:

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this.

The cffi docs mention that the "new callback mechanism" does not have this problem. We only have 2 callbacks, so we might be able to fix this. I'll look into this.

cc @Korijn @berendkleinhaneveld @marcdownie @SuperSimon81 @anentropic

[berendkleinhaneveld]

I read in one of the linked issues that the libffi from homebrew could be troublesome. So I tried removing libffi but ended up removing both my x86 and arm installations of brew completely because for whatever reason I could not get pyenv to build and install a python version anymore :/

I now have only one brew installation (arm64) where I just installed the following packages:
openssl readline sqlite3 xz zlib pyenv (the first few packages are needed to compile python with pyenv).
After that, I installed python 3.9.7, updated pip, installed poetry and ran poetry install (with my local pyproject file which just replicates dev-requirements.txt).

Guess what: I ran the test suite with a lot less problems (I did add the changes of @marcdownie)! It is not perfect yet (there is still quite some stuff failing) but at least cffi seems to work this way.

[berendkleinhaneveld]

I can confirm that having libffi installed with brew was not the culprit... I am now restoring x86 brew installation to see if things keep working.

[berendkleinhaneveld]

Yep, the second brew installation seems to be the reason why I got the error with cffi...

Edit: weird: libffi is not even installed in the x86 version of brew... Let's see if just removing x86 brew makes cffi work correctly.
Edit: Yep, removing x86 brew installation fixed it 🤦

[Korijn]

So could the order in which the linker iterates through the matching binaries be the source of the issue?

[berendkleinhaneveld]

Yeah, I guess. I could try to find out how the linker locates the binaries to see if I can fix it with the right configuration.

[berendkleinhaneveld]

Ok, here is what I know:
I found the source of setup.py here. They check if the path /usr/local/bin/brew (the x86 version) exists to modify the PKG_CONFIG_PATH by appending the brew --prefix libffi path.

Apparently, linking to the libffi from brew on arm64 is not a good idea because it throws the ImportError. It is possible to provide PKG_CONFIG_PATH, but if the libffi package is installed for the arm64 brew, it will be extended with it. By default, the following is added as extra compile arg on darwin: '-iwithsysroot/usr/include/ffi'. So by setting env var PKG_CONFIG_PATH to /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ffi should enable linking with the system default libffi, which should pose no problems.

Gonna try now! 🤞

[berendkleinhaneveld]

No luck... The weird thing is that I don't see my env var show up in the generated clang command:

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/opt/homebrew/Cellar/libffi/3.3_3/include -I/Users/cg/Library/Caches/pypoetry/virtualenvs/wgpu-5bSf_T1V-py3.9/include -I/Users/cg/.pyenv/versions/3.9.7/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c c/_cffi_backend.c -o build/temp.macosx-11.6-arm64-3.9/c/_cffi_backend.o -iwithsysroot/usr/include/ffi
clang -bundle -undefined dynamic_lookup -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/cg/.pyenv/versions/3.9.7/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/cg/.pyenv/versions/3.9.7/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib build/temp.macosx-11.6-arm64-3.9/c/_cffi_backend.o -L/opt/homebrew/Cellar/libffi/3.3_3/lib -lffi -o build/lib.macosx-11.6-arm64-3.9/_cffi_backend.cpython-39-darwin.so

Build command I used was:

PKG_CONFIG_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ffi poetry run pip install -vvv cffi --no-binary :all:

I've also tried to export the PKG_CONFIG_PATH var, but to no avail.

[Korijn]

Maybe this is silly, but PKG_CONFIG_PATH appears to point to an executable called pkgconfig. I'm not sure you are supposed to use it like that?

[berendkleinhaneveld]

You probably have a point. But then I'm out of ideas of what to specify there. I can't find any pkgconfig file anywhere in the macos sdk. Probably would be best to be able to specify whether to link with brew version and open a PR or issue with cffi.

For now I can work around it by either not putting brew on my path or removing libffi from my arm64 brew packages.

[Korijn]

You probably have a point. But then I'm out of ideas of what to specify there. I can't find any pkgconfig file anywhere in the macos sdk. Probably would be best to be able to specify whether to link with brew version and open a PR or issue with cffi.

For now I can work around it by either not putting brew on my path or removing libffi from my arm64 brew packages.

As far as I can see they are asking brew for the path to libffi via brew --prefix libffi. Maybe you can adjust PATH to ensure your arm64 brew executable is preferred?

Also.. the pkgconfig file appears to live under the libffi prefix folder if I'm reading this correctly.

[anentropic]

def use_pkg_config():
    if sys.platform == 'darwin' and os.path.exists('/usr/local/bin/brew'):
        use_homebrew_for_libffi()

this will skip using the homebrew libffi for Arm users because the homebrew path is /opt/homebrew/bin/brew

maybe the problem partly comes if you also have an x64 homebrew installed (e.g. to run in a copy of terminal app using Rosetta)... There were some blog posts advising a setup like that when the M1 Mac first came out, when a lot of things didn't work under Arm yet (I started out that way too but since got rid of the x64 stuff and just use the Arm native homebrew now)

[berendkleinhaneveld]

Maybe you can adjust PATH to ensure your arm64 brew executable is preferred?

The right brew is selected. And linking to the brew version instead of system makes cffi fail.
When I remove the x86 version (/usr/local/...), setup.py will believe I don't have brew (although I still have brew arm64 in /opt/brew) and use the system libffi, which works.
🙃

[berendkleinhaneveld]

Problem 1: building cffi and linking to arm brew libffi results in failure (I don't know how to fix this)

Problem 2: code in setup.py of cffi has a hardcoded path check to default x86 version to see if homebrew is installed (and will link to brew version of libffi, if present) (I can fix this and it provides me a work-around(s) to bypass linking to libffi)

[anentropic]

fwiw:

$ brew info libffi
libffi: stable 3.3 (bottled), HEAD [keg-only]
Portable Foreign Function Interface library
https://sourceware.org/libffi/
/opt/homebrew/Cellar/libffi/3.3_3 (17 files, 617.6KB)
  Poured from bottle on 2021-05-08 at 21:51:12
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/libffi.rb
License: MIT
==> Options
--HEAD
	Install HEAD version
==> Caveats
libffi is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

For compilers to find libffi you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/libffi/include"

For pkg-config to find libffi you may need to set:
  export PKG_CONFIG_PATH="/opt/homebrew/opt/libffi/lib/pkgconfig"

==> Analytics
install: 222,281 (30 days), 616,713 (90 days), 3,250,154 (365 days)
install-on-request: 6,704 (30 days), 19,070 (90 days), 130,463 (365 days)
build-error: 0 (30 days)

$ brew upgrade pyenv
$ pyenv install 3.10.0
$ pyenv virtualenv 3.10.0 cffi-temp
$ pyenv activate cffi-temp

$ pip install wheel==0.34.2
$ pip install cffi
$ pip install wgpu

this much all works for me (on M1) without errors, and I can import wgpu in a python shell

I also don't get the error shown here https://foss.heptapod.net/pypy/cffi/-/issues/506 where pip install bcrypt fails - seems to install fine and I can import bcrypt

where it breaks down is if I then copy the files from the wgpu examples/ folder into my working dir and run one...

$ pip install glfw
...
Successfully installed glfw-2.3.0

$ python triangle_glfw.py
Traceback (most recent call last):
  File "/Users/anentropic/Documents/Dev/Personal/octemp/triangle_glfw.py", line 11, in <module>
    import wgpu.backends.rs  # noqa: F401, Select Rust backend
  File "/Users/anentropic/.pyenv/versions/cffi-temp/lib/python3.10/site-packages/wgpu/backends/rs.py", line 44, in <module>
    from .rs_ffi import ffi, lib, check_expected_version
  File "/Users/anentropic/.pyenv/versions/cffi-temp/lib/python3.10/site-packages/wgpu/backends/rs_ffi.py", line 85, in <module>
    lib = ffi.dlopen(get_wgpu_lib_path())
  File "/Users/anentropic/.pyenv/versions/cffi-temp/lib/python3.10/site-packages/wgpu/backends/rs_ffi.py", line 74, in get_wgpu_lib_path
    raise RuntimeError(f"Could not find WGPU library in {embedded_path}")
RuntimeError: Could not find WGPU library in /Users/anentropic/.pyenv/versions/cffi-temp/lib/python3.10/site-packages/wgpu/resources/libwgpu-release.dylib

this is the error I noted on the other issue #136 (comment)

Sorry, I don't know if I've added anything to the discussion here

[Korijn]

RuntimeError: Could not find WGPU library in /Users/anentropic/.pyenv/versions/cffi-temp/lib/python3.10/site-packages/wgpu/resources/libwgpu-release.dylib

@anentropic Looks like you've used pip to install wgpu, which is good. However, the macos arm64 wheels aren't on pypi yet, so you got an sdist (source distribution). Those don't include the binaries. That's to be expected, sort of.

I could try to make setup.py execute the binary downloading process so that sdists should also "just work"...

I'm working on providing the wheels you need in #194 though!

[Korijn]

Problem 1: building cffi and linking to arm brew libffi results in failure (I don't know how to fix this)

If you build against the x86 brew libffi, and then link to the arm brew libffi at runtime (or the other way around), failure is expected. So the way to fix it would be to ensure you're building and linking against the same libffi.

[Korijn]

I noticed that there is a pre-release on pypi that may have "better" cffi wheels... can anyone try those?

[berendkleinhaneveld]

I noticed that there is a pre-release on pypi that may have "better" cffi wheels... can anyone try those?

Yes, it works! I also notice that arm wheels are available now for macOS, so no local builds are performed now. Shall I bump the version in setup.py or shall we wait for the official release? It seems that 1.15 also supports python 3.6 and up, like us.

[Korijn]

I noticed that there is a pre-release on pypi that may have "better" cffi wheels... can anyone try those?

Yes, it works! I also notice that arm wheels are available now for macOS, so no local builds are performed now. Shall I bump the version in setup.py or shall we wait for the official release? It seems that 1.15 also supports python 3.6 and up, like us.

WOOHOO! Please bump the minimum required version!

[marcdownie]

I believe that the prerelease cffi has fixed the MemoryError and I now have a running triangle_glfw.py on a machine that's never had a x64 brew or conda installed on it.

[almarklein]

I did not expect the memoryerror to disappear so easily, but good news! :)

[almarklein]

Ok in #136 I read that its not crystal clear yet. I made #197 to track the memoryerror/callback. Closing this now.

[anentropic]

Yep, examples are working for me now if I checkout latest src and follow https://github.com/pygfx/wgpu-py#developers 👍

[AdityaHPatwardhan]

I was facing a similar error while building cffi on MAC with M1 chip.
I was trying to install cffi using pip3 install cffi with an environment that used python 3.8
It was failing with failed to build wheel error.

But then I used environment that uses python 3.11 and somehow the issue got fixed. Just commenting a possible alternative here in case someone faces similar issue long after this issue is closed.