Import error with onnx-1.7.0 in mac os catalina

[shonigs]

Hi I get the following error when I import onnx (python 3.7.7)

[libprotobuf ERROR google/protobuf/descriptor_database.cc:394] Invalid file descriptor data passed to EncodedDescriptorDatabase::Add().
[libprotobuf FATAL google/protobuf/descriptor.cc:1356] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 

with onnx-1.6.0 no import errors

I tried changing version of protobuf
using either

pip install protobuf==3.8.0 
pip install protobuf==3.12.2

and
brew reinstall protobuf
didn't help
Any Ideas? I have been trinying to fix this all day

[askhade]

@chinhuang007 : Can you take a look at this issue?

[chinhuang007]

@askhade sorry, I am not a mac user. I will see what I can do and update.

[chinhuang007]

@askhade @shonigs I just tried on a new Mac, with python 3.7.7, protobuf 3.12.2, onnx 1.7.0, and it works fine.

(venv) Chins-MBP:onnx chinhuang$ python
Python 3.7.7 (default, Jun 8 2020, 17:29:50)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import onnx;
exit();
(venv) Chins-MBP:onnx chinhuang$ pip list | grep protobuf
protobuf 3.12.2

[jcwchen]

I tried to install on my Mac with Python 3.7.7 and onnx 1.7.0.
Both protobuf 3.8.0 and protobuf 3.12.2 work for me as well.

But I believe it should be a protobuf's issue.
Try to upgrade your pip, uninstall onnx/protobuf and reinstall them again.
Using protobuf 3.8.0 might be better because I saw some people encounter the same problem with higher protobuf version on their Macs.

Hope it helps. Thanks.

[yoavz]

I've also faced this problem on my mac with onnx==1.7.0, protobuf==3.12.2, Python 3.7.7 on Max OSX Catalina 10.15.5.

It went away when I downgraded to onnx==1.6.0, not sure why.

[jcwchen]

@shonigs @yoavz There are more discussions here #2940 which might be helpful.

[rpatel72]

I've also faced this problem on my mac with onnx==1.7.0, protobuf==3.12.2, Python 3.7.7 on Max OSX Catalina 10.15.5.

It went away when I downgraded to onnx==1.6.0, not sure why.

Thank you, @yoavz. This worked perfectly for me as well.

[dreamibor]

To be clear, the key here is to compile and install ONNX with a much lower version of protobuf. For example:

conda install -c conda-forge protobuf numpy

git clone https://github.com/onnx/onnx.git
cd onnx
git submodule update --init --recursive
python setup.py install

Then we can install TensorFlow as normal, which will also upgrade the version of protobuf.

conda install -c conda-forge protobuf installs protobuf==3.4.0 on my Mac and it's more or less equivalent to
pip install protobuf==3.4.0.

The information here is not clear, installing from source with a higher version of protobuf won't work.

[jcwchen]

Hi @dreamibor,
I am still using macOS Catalina (10.15.7), but I cannot repro this error anymore... I tried many kinds of protobuf versions and all worked even in the conda environment.
May I know what your pip show protobuf/protoc --version/macOS version/ONNX version are in your failed cases? Thank you.

[dreamibor]

This is the output of runninng pip show protobuf --version:

Name: protobuf
Version: 3.14.0
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: None
Author-email: None
License: 3-Clause BSD License
Location: /Users/xxx/Downloads/Tools/test_onnx/lib/python3.8/site-packages
Requires: six
Required-by: tensorflow, tensorboard, onnx

I tested with a clean virtual environment by running:

python -m venv test_onnx
source test_onnx/bin/activate

pip install onnx
pip install tensorflow

python3 -c 'import tensorflow, onnx'

And it will fail on MacOS Catalina 10.15.7, the ONNX version is 1.8.0 installed through pip, and my Python version is 3.8.3.

[jcwchen]

Thank you for providing the info. I can repro this now.
There are 3 possible ways to workaround this:

1. Like you mentioned, downgrade the protobuf version.
2. Use ONNX 1.6.0.
3. import onnx before import tensorflow

I have 2 questions:

1. How could you use protobuf==3.4.0 with the latest tensorflow? (I saw it has a requirement: tensorflow 2.4.0 requires protobuf>=3.9.2)
2. I thought it only happens when import tensorflow first then import onnx. Did you encounter this error in other scenarios?

Thank you for bringing this up again to remind that this issue is still not resolved.

[dreamibor]

Hi @jcwchen,

1. Protobuf ==3.4.0 is only used to compile and install ONNX, and then when I install TensorFlow, pip will install a newer version of protobuf such as 3.14.0, to be compatible with TensorFlow. That's why I emphasised here the order of installation is important.
2. I think that's my case as well, importing TensorFlow and then ONNX will cause the error.

Thanks

[alvercau]

I cannot get it to work, after trying several combinations of protobuf, onnx and tensorflow...
I have python 3.8 and macOS Catalina version 10.15.6. I've tried installing onnx with pip, from source, before tensorflow, after tensorflow, with several versions of protobuf, the issue remains. Can someone who got this to work specify exactly which combination of versions works? I am stuck with python 3.8 and tensorflow 2.2, but am flexible for onnx versions.

[jcwchen]

Hi @alvercau,
Does pip install onnx==1.6.0 work for you?

[alvercau]

I finally got it to work! For one reason or another, everything works fine in a conda environment (I was using a Virtualenv environment before).

If installed in this order:

conda create --name myenv python=3.8

conda activate myenv

conda install -c conda-forge onnx

pip install tensorflow==2.2.1

everything works as it should.

[dreamibor]

What's ONNX version you installed by using conda?

[alvercau]

What's ONNX version you installed by using conda?

It is version 1.7.0

[jcwchen]

Thanks @alvercau for finding conda does work. It makes sense to me because the ONNX wheel is built by MacOS 10.15 in conda. (https://github.com/conda-forge/onnx-feedstock/blob/master/.azure-pipelines/azure-pipelines-osx.yml)

Provide another workaround for users who want to use the latest ONNX:
Build ONNX from source with MacOS 10.15+ version.

[alvercau]

building from source didn't work for me... I kept on getting the libprotobuf error.

[jcwchen]

building from source didn't work for me... I kept on getting the libprotobuf error.

Is your Mac 10.15? I saw the above message. Never mind. Thanks.

[jcwchen]

Hi @alvercau,
If you have time, could you please help me to verify this package in PyPi (not in conda)? https://github.com/onnx/onnx/suites/1868153005/artifacts/36372267
It's an ongoing ONNX 1.8.1 package which is built from Mac 10.15. It does resolve my libprotobuf issue but I am afraid it does not resolve yours. Then I will need to target other root cause. Please pip uninstall onnx several times until there is no existing ONNX package before pip install onnx-1.8.0-cp38-cp38-macosx_10_15_x86_64.whl. Thank you!

[jcwchen]

I believe this problem should be resolved in ONNX 1.8.1. Please let me know if ONNX 1.8.1 still doesn't work from your end. Thank you all!

[dreamibor]

@jcwchen I can confirm on my Mac, the problem is gone with ONNX 1.8.1
OS: MacOS Big Sur 11.1
Python: 3.8.5
ONNX: 1.8.1
TensorFlow: 2.4.1
protobuf: 3.14.0

[jcwchen]

@dreamibor Thank you so much for the feedback! Close this issue now. Please let me know if this issue still exists with your ONNX 1.8.1 and I will reopen this issue. Thank you.