grpcio fails to install on Apple Silicon

[yoav-orca]

What version of gRPC and what language are you using?

grpcio-1.34.0

What operating system (Linux, Windows,...) and version?

macOS big Sur 11.0.1, running on M1 Apple Silicon

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8.7 (default, Dec 30 2020, 02:09:32)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

$ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: arm64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

pip install grpcio==1.34.0

What did you expect to see?

That grpcio will be successfully installed

What did you see instead?

Collecting grpcio
  Using cached grpcio-1.34.0.tar.gz (21.0 MB)
    ERROR: Command errored out with exit status 1:
     command: /Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"'; __file__='"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-pip-egg-info-z41yqiw6
         cwd: /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/
    Complete output (10 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py", line 359, in <module>
        if mac_target and (pkg_resources.parse_version(mac_target) <
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 113, in parse_version
        return packaging.version.Version(v)
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
        match = self._regex.search(version)
    TypeError: expected string or bytes-like object
    ASM Builds for BoringSSL currently not supported on: macosx-11-arm64
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

[Tenzer]

I managed to get the grpcio Python package installed on Python 3.9.1 installed via Pyenv on a M1 macOS:

Python 3.9.1 (default, Jan  5 2021, 10:39:36)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Native arm64:

$ lipo -archs $(which python)
arm64

I had to set a few environment variables to get it to work:

GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

to make the installation process use the libraries I've already got on the system, and then the following to point it at OpenSSL installed via Homebrew:

CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

[ebitgeek]

you can use system python3 (/usr/bin/python3)

mkdir grpctest
cd grpctest
/usr/bin/python3 -m pip install virtualenv
/usr/bin/python3 -m virtualenv venv
source venv/bin/activate __
arch -x86_64 pip install --upgrade pip setuptools
arch -x86_64 pip install grpcio
arch -x86_64 pip install grpcio-tools
touch test.proto
...
arch -x86_64 python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. test.proto
touch main.py
arch -x86_64 python main.py

attention

arch -x86_64 should be used before the command

[lidizheng]

@yoav-orca We currently doesn't have ARM64 machines to produce binary wheels, but we are working on a plan.

Back to the build, the failure is not caused by the ASM optimization. Can you check the result of sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') on your Mac? It seems returning a number or some sort, which supposed to be a string (or bytes) of targeted macOS version.

EDIT: if none presented, you could try to set it to your macOS version, e.g. Big Sur is 11.0.

[Tenzer]

I just tried to install the library from the master branch with the fix included, but it still fails out of the box.

Is the version of BoringSSL and zlib referenced in this repository supposed to be working on ARM macOS? If not it might be better to try and detect when running on a ARM Mac (platform.machine() can do this) and set the equivalent of the following environment variables automatically:

GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Does that sound reasonable? If so, I don't mind trying to put a PR together to do that.

[lidizheng]

@Tenzer That would be great if you can put up a PR, and you can ensure it works on the target machine (I don't have an Apple M1).

[achimnol]

I got a successful build using @Tenzer 's solution, with addition of CFLAGS and LDFLAGS using brew --prefix protobuf, brew --prefix openssl, brew --prefix zlib, and setting the GRPC_PYTHON_BUILD_SYSTEM_* environment variables, with arm64 homebrew and Python 3.9.1 built via pyenv's latest master.

[dvanmali]

Thanks @Tenzer and @achimnol! This worked for me as well.

Note if the following issue occurs after those recommendations:

File "____/venv/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

Edit version.py line 275 on your system to:

match = self._regex.search(str(version))

[doublecc161]

Setting the environment variables as recommended by @Tenzer works perfectly for me, too! Thank you! I used it to install stable_baselines3 on my Apple Silicon M1 🙂

I simply set the environment variables first:

export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Then installed the grpcio package with:

pip install grpcio

At the time of writing this installs and works with grpcio-1.35.0

@lidizheng @Tenzer Since it seems like this problem is not yet resolved I would offer help creating a PR, however I am completely new to this kind of stuff (pip install, setuptools etc.) and don't really know where I would have to put the code. This is what I think would have to go somewhere according to your contributions:

import platform
import os
from distutils import util

if 'arm64' in platform.machine() and 'mac' in util.get_platform():
    os.environ["GRPC_PYTHON_BUILD_SYSTEM_OPENSSL"] = "1"
    os.environ["GRPC_PYTHON_BUILD_SYSTEM_ZLIB"] = "1"
else:
    pass

Maybe in setup.py around line 82? Of course the required import statements are there already.

My Setup

Device: Apple M1 MacBook Pro running macOS Big Sur 11.2
Python:

Python 3.9.1 | packaged by conda-forge | (default, Jan 26 2021, 01:30:54) 
[Clang 11.0.1 ] on darwin

Let me know if there is any more info that could be helpful!

I also added the wheel generated by pip if that is helpful to anyone.
grpcio-1.35.0-cp39-cp39-macosx_11_0_arm64.whl.zip

[andreaestrada]

I tried doing many of the above, and unfortunately all failed for me. Ended up getting it to work by following the steps here (but for grpcio) https://stackoverflow.com/questions/65564673/how-do-i-import-pandas-on-apple-m1-chip. Posting in case it's helpful to anyone in the future.

[finnatsea]

Still having this problem after having tried the above. Also, the listed solutions are pretty involved workarounds. Is someone working on a more permanent fix?

[dgaedcke]

Yes, someone please respond ... I've already burned over 2 days on this error!!

I've tried Homebrew w Py3.8, Py3.9 and Conda-miniforge with both as well
I've tried bash arm64, bash x86, zsh arm64

I've even tried the Panda's example above but it didn't solve the problem for me...

How do I install and run this library?? The failure stacktrace is below:

import google.cloud.ndb as ndb
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/google/cloud/ndb/init.py", line 28, in
from google.cloud.ndb.client import Client
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/google/cloud/ndb/client.py", line 18, in
import grpc
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/init.py", line 23, in
from grpc._cython import cygrpc as _cygrpc
ImportError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 2): Symbol not found: _CFRelease
Referenced from: /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so
Expected in: flat namespace
in /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so

[lidizheng]

@dgaedcke I don't have a M1 machine. The answer is based on what I read.

Have you try to build your environment with emulated x86?

arch -x86_64 /bin/bash

Then you should be able to use x86 Python, and x86 grpcio binary wheel, no compilation needed.

[dgaedcke]

Thanks for your response.
Your suggestion is a really good one!!!
And I already got this fixed by using the Conda miniforge build of grpcio
I suspect this will be fixed in this repo when they release 1.39.1 but I'm not sure!!
really appreciate your help!

[Korolen]

I couldn't easily get a build process working on my Monterey M1, but the binary worked.

Details:

Setting the two environment variables still left me with

Traceback (most recent call last):
  File "/Users/tom/code_tmp/clone/grpc/examples/python/helloworld/./greeter_server.py", line 19, in <module>
    import grpc
  File "/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace (_CFRelease)

when trying to import grpc.

Maybe "If we export magic variables from the conda recipe, keep conda libraries installed and install Cython, then" would have worked, but I don't understand this enough to do it confidently (keep what installed how?). I haven't read the whole thread.

conda install -c conda-forge grpcio gave me

(basemod) tom@MacBook-Pro helloworld % conda install grpcio -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 125, in _convert
        return self.names[name]
    KeyError: 'conda-forge/osx-arm64::certifi-2022.6.15-py38h10201cd_0'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__
        return func(*args, **kwargs)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
        exit_code = do_call(args, p)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 90, in do_call
        return getattr(module, func_name)(args, parser)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/install.py", line 259, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 313, in solve_final_state
        ssc = self._add_specs(ssc)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 605, in _add_specs
        conflict_specs = ssc.r.get_conflicting_specs(tuple(concatv(
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/resolve.py", line 1109, in get_conflicting_specs
        C = r2.gen_clauses()
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/io.py", line 86, in decorated
        return f(*args, **kwds)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/resolve.py", line 911, in gen_clauses
        nkey = C.Not(self.to_sat_name(prec))
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 144, in Not
        return self._eval(self._clauses.Not, (x,), (), polarity, name)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 130, in _eval
        args = self._convert(args)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 120, in _convert
        return type(x)(map(self._convert, x))
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 127, in _convert
        raise ValueError("Unregistered SAT variable name: {}".format(name))
    ValueError: Unregistered SAT variable name: conda-forge/osx-arm64::certifi-2022.6.15-py38h10201cd_0

`$ /Users/tom/opt/anaconda3/bin/conda install grpcio -c conda-forge`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=basemod
                CONDA_EXE=/Users/tom/opt/anaconda3/bin/conda
             CONDA_PREFIX=/Users/tom/opt/anaconda3/envs/basemod
    CONDA_PROMPT_MODIFIER=(basemod)
         CONDA_PYTHON_EXE=/Users/tom/opt/anaconda3/bin/python
               CONDA_ROOT=/Users/tom/opt/anaconda3
              CONDA_SHLVL=1
           CURL_CA_BUNDLE=<not set>
                 INFOPATH=/opt/homebrew/share/info:
                  MANPATH=/Users/tom/.nvm/versions/node/v17.6.0/share/man:/Users/tom/software/gi
                          t-subrepo/man:/opt/homebrew/share/man::
                     PATH=/Users/tom/opt/anaconda3/bin:/Users/tom/opt/anaconda3/envs/basemod/bin
                          :/Users/tom/opt/anaconda3/condabin:/Users/tom/.nvm/versions/node/v17.6
                          .0/bin:/Users/tom/.poetry/bin:/opt/homebrew/opt/openssl@1.1/bin:/Users
                          /tom/software/git-subrepo/lib:/Users/tom/Library/Jupyter/kernels:/User
                          s/tom/Documents/code/mon/scripts:/Users/tom/.nimble/bin:/Users/tom/ins
                          tall/nim/bin:/Users/tom/Applications/Sublime Text.app/Contents/SharedS
                          upport/bin:/Users/tom/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/lo
                          cal/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech
                          Preview.app/Contents/Public:/usr/local/zfs/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : basemod
    active env location : /Users/tom/opt/anaconda3/envs/basemod
            shell level : 1
       user config file : /Users/tom/.condarc
 populated config files : /Users/tom/.condarc
          conda version : 4.13.0
    conda-build version : 3.21.9
         python version : 3.9.12.final.0
       virtual packages : __osx=12.5.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/tom/opt/anaconda3  (writable)
      conda av data dir : /Users/tom/opt/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/anaconda/osx-arm64
                          https://conda.anaconda.org/anaconda/noarch
          package cache : /Users/tom/opt/anaconda3/pkgs
                          /Users/tom/.conda/pkgs
       envs directories : /Users/tom/opt/anaconda3/envs
                          /Users/tom/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.9.12 Darwin/21.6.0 OSX/12.5.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Installing the binary wheel from the latest artifact at https://github.com/pietrodn/grpcio-mac-arm-build/actions by downloading it and

pip uninstall grpcio
cd ~/Downloads/artifacts/
pip install ./grpcio-1.48.1-cp39-cp39-macosx_11_0_arm64.whl

worked a treat. Thank you so much!! <3

[jrgoulet]

I ran into the same problem today with the following specs:

python version: 3.10.6 grpcio version: 1.49.1 os: mac os monterey 12.6 (M1 PRO)

Solved it by uninstalling, cleaning the conda and pip cache and then reinstalling grpcio again:

pip uninstall grpcio
conda clean --all
pip cache purge
pip install grpcio

Confirming python 3.10.6 works on Monterey 12.5 (M1 Pro) with no env vars set. 3.9 did not seem to work for me with any of the incantations above.

[elementljarredc]

Would it be fair to say that this issue is not resolved? Should there be another one opened, or this one perhaps? We've got OSS users that encounter this problem somewhat regularly

[edvard-bjarnason]

Of course the issue should be open, it is not fixed. I don't understand the trend on GitHub to close issues before a fix has been created, it is not like the issue goes away just by marking the issue closed:( Now it is almost 2 years since the issue was created and nobody cares enough about this to actually do the work needed to fix it. It is really frustrating that an important package like this can not be installed by pip without problems.

On my Mac m1 12.6 python 3.10.6 I was not able to install 1.50.0 I simply gave up trying get it compiled :( and installed 1.46.3 with the following env vars set:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
GRPC_BUILD_WITH_BORING_SSL_ASM=
GRPC_PYTHON_BUILD_SYSTEM_CARES=True
GRPC_PYTHON_USE_PREBUILT_GRPC_CORE=
GRPC_PYTHON_BUILD_WITH_CYTHON=True
GRPC_PYTHON_LDFLAGS= -framework CoreFoundation

Not sure if the env vars matter though

[edvard-bjarnason]

Just to follow up on my previous comment, it turned out I could not use version 1.46.3. Anyway I managed to install version grpcio version 1.50.0. My pyenv was pulling in libraries for macOS 12.4 and after I updated pyenv and reinstalled python versions 3.10.6 (which pyenv builds from source) I could install grpcio version 1.50.0.

Here are the commands I used:
brew upgrade pyenv
pyenv uninstall 3.10.6
pyenv install 3.10.6
pyenv virtualenv 3.10.6 google
pyenv activate google
pip install grpcio
...
Installing collected packages: six, grpcio
Running setup.py install for grpcio ... done
Successfully installed grpcio-1.50.0 six-1.16.0

[Ambassador4ik]

Would it be fair to say that this issue is not resolved? Should there be another one opened, or this one perhaps? We've got OSS users that encounter this problem somewhat regularly

Yeah it is absolutely true, I spent like 2 hours trying to install it without venv, nothing really helps...

[pietrodn]

Hello, since Google is doing nothing about this, I decided to set up a periodic GitHub Actions workflow to provide daily builds of Apple Silicon wheels for Mac. It will automatically cut a new release the day a new grpcio version is released on PyPI.

In the releases page of this repo you can find and download the wheels for Python 3.8, 3.9, 3.10, 3.11.
(Use at your own risk, these are not tested.)
https://github.com/pietrodn/grpcio-mac-arm-build

[maxkuangsy]

I found out that setuptools-65.5.0 seems not able to install the latest grpcio-2.10.0, so I choose to install grpcio2.9.0 and then it works.

[sanghunka]

@pietrodn Thanks 👍👍

[techdragon]

@pietrodn the issues appear to also affect the grpcio-tools package can you add that to your build example (and the repo where you've got working builds)

[pietrodn]

@techdragon ✅ Done! https://github.com/pietrodn/grpcio-mac-arm-build/releases

[levrik]

Official macOS ARM builds seem to land in GRPC 1.52: #31747

Also GRPC 1.51 builds in under a minute from source for me on a M1 without any env variables set.

[gnossen]

@levrik Yes, to confirm, 1.52.0 includes prebuilt universal2 artifacts which work on both x86-64 and arm64 macs. The release candidate is already uploaded and the official release is scheduled for later this week.

[gnossen]

1.52.0 has been released with proper universal2 artifacts which support both x86 macs and M1/2 macs.

[legioz]

1.52.0 has been released with proper universal2 artifacts which support both x86 macs and M1/2 Macs.

1.52.0 is Yanked

[sontek]

Can we re-open?

[marns93]

Can we re-open?

I can see that there is a new release https://github.com/grpc/grpc/releases/tag/v1.52.1 and that was announced for today.
#32306 (comment)

But it is not published on PyPI....

[gnossen]

1.51.2 is an unrelated C++-only patch release for a security vulnerability affecting only the C++ API. The release process for 1.51.3, which backports universal2 artifacts, is in progress and is still estimated to be complete today.

[gnossen]

Due to an issue with the CI build, the estimate is now Tuesday, February 28th.

[gnossen]

1.51.3 has now been released with universal2 artifacts. I'll leave this issue open for a couple more days in case there are further issues.