Python - Mac M1 - ImportError: cannot import name '_message' from 'google.protobuf.pyext'

[Laurens-T]

What version of protobuf and what language are you using?
Version: 3.17.3
Language: Python

What operating system (Linux, Windows, ...) and version?
Mac0S Big Sur 11.4, M1 version

What runtime / compiler are you using (e.g., python version or gcc version)
Python 3.9.5

gcc:

Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.9)
Target: arm64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What did you do?

➜  python --version
Python 3.9.5

➜  python -m venv venv
➜  source venv/bin/activate

(venv) ➜  pip install -U pip protobuf
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1)
Collecting pip
  Using cached pip-21.1.3-py3-none-any.whl (1.5 MB)
Collecting protobuf
  Using cached protobuf-3.17.3-py2.py3-none-any.whl (173 kB)
Collecting six>=1.9
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, protobuf, pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-21.1.3 protobuf-3.17.3 six-1.16.0
(venv) ➜  python -c "from google.protobuf.pyext import _message"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/Users/home/Programming/python/protobuf/venv/lib/python3.9/site-packages/google/protobuf/pyext/__init__.py)

What did you expect to see
A successful import.

What did you see instead?
ImportError: cannot import name '_message' from 'google.protobuf.pyext'

Anything else we should know about your project / environment
No

Other things I've tried
I have tried running the same snippet inside native and emulated docker containers. I could successfully install protobuf in the emulated container.

# The relevant protobuf file:
FROM python:3.9.5

RUN pip install -U pip protobuf
CMD ["python", "-c", "from google.protobuf.pyext import _message"]

➜ docker build -t protobuf-arm .
    [+] Building 2.8s (7/7) FINISHED
    ...

➜ docker run --rm -it protobuf-arm
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/usr/local/lib/python3.9/site-packages/google/protobuf/pyext/__init__.py)

➜ export DOCKER_DEFAULT_PLATFORM=linux/amd64 && docker build -t protobuf-amd .
    [+] Building 2.0s (6/6) FINISHED
    ...

➜  protobuf docker run --rm -it protobuf-amd
➜

EDIT: This issue persists for protobuf version 3.18.0.

[Laurens-T]

Some of my colleagues also recently received M1 macs, and I asked them to follow the same steps to install protobuf locally.
One of them had the same error as me, the other one did manage to execute the final step without any issues.

I have no idea what the difference in setups is exactly.

[TeoZosa]

I had the same problem and discovered it was because required shared libraries are missing from the -py2.py3-none-any.whl pure python wheel that gets pulled in automatically.

I was able to solve it by building the protobuf wheel from source:

brew install protobuf
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"
# In my virtual environment: 
pip install protobuf=="$(brew list --version protobuf | awk '{print $2}')" --install-option="--cpp_implementation"
# If the pure python protobuf wheel is already installed, run this instead:
# pip install protobuf=="$(brew list --version protobuf | awk '{print $2}')" --force-reinstall --no-deps --install-option="--cpp_implementation"

Hope this helps!

Edit (2021/11/09):

• Note that this installs the version of the protobuf Python library that matches the version of the pre-compiled protoc binary that was installed via brew (brew install protobuf).
• If you need a newer version than what brew provides, you're going to have to build protoc from source. Assuming you already have the pure Python protobuf wheel installed, this would look something like:

# Download and unpack protobuf compiler source code
PROTOBUF_RELEASES_URL="https://github.com/protocolbuffers/protobuf/releases/download"
TARGET_PROTOBUF_VERSION=$(pip show protobuf | grep Version | sed -r 's/Version: (.*)/\1/') # get the version of the (broken) protobuf wheel currently installed
PROTOC_SRC_ARCHIVE="protobuf-cpp-${TARGET_PROTOBUF_VERSION}.tar.gz"
curl -sSL "${PROTOBUF_RELEASES_URL}/v${TARGET_PROTOBUF_VERSION}/${PROTOC_SRC_ARCHIVE}" | tar -C /tmp -xzf -

# Build protobuf compiler from source (this will take a while)
#   see: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md
PROTOC_SRC_PATH="/tmp/protobuf-${TARGET_PROTOBUF_VERSION}" && \
pushd "${PROTOC_SRC_PATH}" && \
./configure && \
make -j8 && \
make check && \
sudo make install && \
popd && \
rm -rf "${PROTOC_SRC_PATH}" # Clean src/build files 

# In your virtual environment:
INSTALL_PREFIX_PATH="/usr/local" && \
export CFLAGS="-I${INSTALL_PREFIX_PATH}/include" && \
export LDFLAGS="-L${INSTALL_PREFIX_PATH}/lib" && \
pip install protobuf=="${TARGET_PROTOBUF_VERSION}" --force-reinstall --no-deps --install-option="--cpp_implementation" 

[Laurens-T]

This solved my problem. Thanks!

[haberman]

We are currently preparing a 4.21.0 release of protobuf which will release with M1-compatible universal2 wheels for Python.

The M1 support will not be in 4.21.0-rc1, but it should be in all subsequent releases.

This is part of a major rework of the C extension which should also deliver some performance improvements. I expect these performance improvements to be especially noticeable on Mac: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

[deannagarcia]

A universal binary has been published for 4.21.0! Feel free to open another bug if this issue still exists in the new version.

[przna]

Import error on M2 Mac. ModuleNotFoundError: No module named 'google.protobuf.pyext._message'

[haberman]

Underscore-prefixed names like _message are not part of the public API, and can break at any time.

More info here: #15643 (comment)