Pynacl on Apple Silicon/ARM64 #654

johnalanwoods · 2021-01-18T15:39:06Z

Trying to pip3 install pynacl on an M1 Mac with pip3 version 20.3.3.

Get this message on install - any ideas?

Other libs are working ok, such as tinyec.

Collecting pynacl
  Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
     |████████████████████████████████| 3.4 MB 1.8 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/tmpyxshsvrm
         cwd: /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-install-48_63pd2/pynacl_34f24b5a9a4045dfbb49726500e88931
    Complete output (39 lines):
    Traceback (most recent call last):
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 216, in <module>
        setup(
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 432, in __init__
        _Distribution.__init__(self, {
      File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 708, in finalize_options
        ep(self)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 715, in _finalize_setup_keywords
        ep.load()(self, ep.name, value)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "src/bindings/build.py", line 36, in <module>
        ffi = FFI()
      File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
      Referenced from: /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
      Expected in: flat namespace
     in /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/tmpyxshsvrm Check the logs for full command output.

This is what I have installed:

Package      Version
------------ -------
meson        0.56.2
pip          20.3.3
protobuf     3.14.0
pycrypto     2.6.1
pycryptodome 3.9.9
setuptools   51.1.1
six          1.15.0
tinyec       0.3.1
wheel        0.36.2

The text was updated successfully, but these errors were encountered:

reaperhulk · 2021-01-18T15:52:30Z

Building pynacl in a venv using M1 native homebrew python3.9 works on my M1 test machine.

This, unfortunately, makes it highly like that your issue is some type of environment problem specific to your circumstances. The error suggests symbol issues in cffi so you may want to try separately installing cffi first, or reinstalling it using --no-binary cffi to make it recompile in case you have a cached wheel compiled against a non-existent dylib. If you're not installing in a venv try making a new one and installing there as well.

ice29 · 2021-01-19T16:30:51Z

I have the same problem on my M1... bcrypt also fails with exactly the same error.
Tried to install cffi or install it with no-binary, no help...

reaperhulk · 2021-01-19T17:52:19Z

What happens if you run this command:

python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:

_ffi_prep_closure is an interesting symbol since it doesn't exist in my symbol table for _cffi_backend.cpython-39-darwin.so and yet no issues occur on my machine.

ice29 · 2021-01-19T17:58:54Z

The same problem comes up, but it is apparently known issue with CFFI
https://foss.heptapod.net/pypy/cffi/-/issues/464#note_147550

other libs have the same problem, e.g.
gevent/gevent#1721

johnalanwoods · 2021-01-19T18:02:25Z

Maybe it's working for @reaperhulk because he's using x86 arch in terminal/rosetta?

❯ file /usr/local/bin/python3                                          
/usr/local/bin/python3: Mach-O 64-bit executable x86_64

❯ file /opt/homebrew/bin/python3                            
/opt/homebrew/bin/python3: Mach-O 64-bit executable arm64

ice29 · 2021-01-19T18:10:00Z

yes, I have just setup rosetta2 x86_64 python3 and it works with that one...

johnalanwoods · 2021-01-19T18:19:57Z

@ice29, I guess it will work in time so...

reaperhulk · 2021-01-19T19:48:13Z

I bought this M1 mac specifically to test against arm64 for PyCA projects so my entire Python toolchain (and everything I use to test pynacl, bcrypt, and cryptography) is M1 native and not run through rosetta 2 😄

❯ file .venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so 
lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: Mach-O 64-bit bundle arm64

Given the amount of state inherent in a C build toolchain it's possible I did something (that I've now forgotten) to make this all work I suppose.

eblume · 2021-05-27T14:48:36Z

@reaperhulk I am unfortunately still getting this error on my m1 mac. I am also in the process of filing a 'downstream' bug report with the Fabric team, as pip install fabric currently fails on installing pynacl as a dependency due to this bug, fabric is a developer dependency in my project which means that I'm not presently able to even install my development environment on m1.

Note: I am NOT using rosetta2.

python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:

Collecting pip
  Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.1.2
Collecting pynacl
  Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /<REDACTED>/bin/python3 /<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmpg27kpjth
         cwd: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-install-o2vytdsa/pynacl_ded61a19ae59488294e578a090ae4d22
    Complete output (39 lines):
    Traceback (most recent call last):
      File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
        main()
      File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 166, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 150, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 216, in <module>
        setup(
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 455, in __init__
        _Distribution.__init__(self, {
      File "/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 801, in finalize_options
        ep(self)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 808, in _finalize_setup_keywords
        ep.load()(self, ep.name, value)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "src/bindings/build.py", line 36, in <module>
        ffi = FFI()
      File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
      Referenced from: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
      Expected in: flat namespace
     in /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz#sha256=54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 (from https://pypi.org/simple/pynacl/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*). Command errored out with exit status 1: /<REDACTED>/tmpvenv/bin/python3 /Users/erich/code/work/Project-Vitamin/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmpg27kpjth Check the logs for full command output.

reaperhulk · 2021-05-27T15:28:49Z

@eblume What happens if you uninstall cffi and then attempt to install it with LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:? The problem you're seeing is a mismatch between the libffi cffi compiled against and the one it's finding at runtime. This typically occurs when it compiles against homebrew and then attempts to load the system libffi.

eblume · 2021-05-27T15:34:09Z

 arm64 :: ~
❯ python --version
Python 3.9.4

 arm64 :: ~
❯ pip uninstall cffi
Found existing installation: cffi 1.14.5
Uninstalling cffi-1.14.5:
  Would remove:
    /Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
    /Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cffi-1.14.5-py3.9.egg-info
    /Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cffi/*
Proceed (y/n)? y
  Successfully uninstalled cffi-1.14.5

 arm64 :: ~
❯ LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:
Collecting cffi
  Using cached cffi-1.14.5.tar.gz (475 kB)
Requirement already satisfied: pycparser in ./.pyenv/versions/3.9.4/lib/python3.9/site-packages (from cffi) (2.20)
Skipping wheel build for cffi, due to binaries being disabled for it.
Installing collected packages: cffi
    Running setup.py install for cffi ... done
Successfully installed cffi-1.14.5

 arm64 :: ~
❯ python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:

Requirement already satisfied: pip in ./tmpvenv/lib/python3.9/site-packages (21.1.1)
Collecting pip
  Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-21.1.2
Collecting pynacl
  Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/erich/tmpvenv/bin/python3 /Users/erich/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmp0jl8_sel
         cwd: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-install-dp3b64jj/pynacl_25a0f9d512d449688f625b9c00c53799
    Complete output (39 lines):
    Traceback (most recent call last):
<... Redacting Traceback to save some space ...>
    ImportError: dlopen(/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
      Referenced from: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
      Expected in: flat namespace
     in /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz#sha256=54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 (from https://pypi.org/simple/pynacl/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*). Command errored out with exit status 1: /Users/erich/tmpvenv/bin/python3 /Users/erich/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmp0jl8_sel Check the logs for full command output.
  Using cached PyNaCl-1.3.0.tar.gz (3.4 MB)
^C  Installing build dependencies ... canceled
ERROR: Operation cancelled by user

I also tried with replacing the last command with simply pip install pynacl - same effect.

bates64 · 2021-08-18T16:44:57Z

I was able to install using the following setup:

brew install libsodium
ln -s /opt/homebrew/Cellar/libsodium/1.0.18_1/lib/libsodium.a /usr/local/lib/libsodium.a
SODIUM_INSTALL=system LIBSODIUM_MAKE_ARGS=-j8 pip install pynacl

carlosonunez · 2021-11-08T00:17:41Z

If the previous solution isn't working for you, add CPPFLAGS="-I /opt/homebrew/Cellar/libsodium/1.0.18_1/include" to the beginning of pip install.

reaperhulk · 2022-01-07T22:48:25Z

We ship a universal2 wheel now.

milosivanovic mentioned this issue Feb 27, 2021

Invalid configuration arm64-apple-darwin20.3.0': machine arm64-apple' not recognized #657

Closed

rkhwaja mentioned this issue Mar 16, 2021

Doesn't install cleanly on M1 macos rkhwaja/fs.onedrivefs#26

Closed

M-Whitaker mentioned this issue Apr 17, 2021

Issues running on Apple Silicon (ARM64 Native) openairplay/airplay2-receiver#24

Closed

eblume mentioned this issue May 27, 2021

Install fails on arm64 (Mac M1, Silicon) due to pynacl issue (via libcffi) fabric/fabric#2164

Closed

ash-cryptocom mentioned this issue Jun 30, 2021

Remove pynacl dependency. paramiko/paramiko#1806

Open

legendofmiracles mentioned this issue Aug 22, 2021

python3Packages.pynacl: doesn't cross-compile NixOS/nixpkgs#135149

Closed

tiagoposse mentioned this issue Nov 30, 2021

support multi-arch docker images and update base to debian bullseye DefectDojo/django-DefectDojo#5528

Closed

reaperhulk closed this as completed Jan 7, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pynacl on Apple Silicon/ARM64 #654

Pynacl on Apple Silicon/ARM64 #654

johnalanwoods commented Jan 18, 2021 •

edited

reaperhulk commented Jan 18, 2021

ice29 commented Jan 19, 2021

reaperhulk commented Jan 19, 2021

ice29 commented Jan 19, 2021 •

edited

johnalanwoods commented Jan 19, 2021 •

edited

ice29 commented Jan 19, 2021

johnalanwoods commented Jan 19, 2021

reaperhulk commented Jan 19, 2021

eblume commented May 27, 2021

reaperhulk commented May 27, 2021

eblume commented May 27, 2021 •

edited

bates64 commented Aug 18, 2021

carlosonunez commented Nov 8, 2021

reaperhulk commented Jan 7, 2022

Pynacl on Apple Silicon/ARM64 #654

Pynacl on Apple Silicon/ARM64 #654

Comments

johnalanwoods commented Jan 18, 2021 • edited

reaperhulk commented Jan 18, 2021

ice29 commented Jan 19, 2021

reaperhulk commented Jan 19, 2021

ice29 commented Jan 19, 2021 • edited

johnalanwoods commented Jan 19, 2021 • edited

ice29 commented Jan 19, 2021

johnalanwoods commented Jan 19, 2021

reaperhulk commented Jan 19, 2021

eblume commented May 27, 2021

reaperhulk commented May 27, 2021

eblume commented May 27, 2021 • edited

bates64 commented Aug 18, 2021

carlosonunez commented Nov 8, 2021

reaperhulk commented Jan 7, 2022

johnalanwoods commented Jan 18, 2021 •

edited

ice29 commented Jan 19, 2021 •

edited

johnalanwoods commented Jan 19, 2021 •

edited

eblume commented May 27, 2021 •

edited