Protobuf version problem with Python

[ghost]

We have a Python project which depends on ortools 6.6.4656. Most of the team are on Ubuntu machines, it works fine for them. I'm on a Mac, running macOS 10.13.3 . Whenever I try to run a script that depends on ortools, the process fails and I get the following output in the console:

[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/stubs/common.cc:79] This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.4.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "ortools/gen/ortools/data/rcpsp.pb.cc".)
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.4.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "ortools/gen/ortools/data/rcpsp.pb.cc".)

The versions of ortools and protobuf I actually have installed:

> pip show ortools
Name: ortools
Version: 6.6.4656
Summary: Google OR-Tools python libraries and modules
Home-page: https://developers.google.com/optimization/
Author: Google Inc
Author-email: lperron@google.com
License: Apache 2.0
Location: /Users/nikolay/.pyenv/versions/3.5.4/envs/main/lib/python3.5/site-packages
Requires: six, protobuf

> pip show protobuf
Name: protobuf
Version: 3.5.2
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /Users/nikolay/.pyenv/versions/3.5.4/envs/main/lib/python3.5/site-packages
Requires: setuptools, six

When I pip install our project dependencies, pip downloads the following wheel for ortools: Downloading ortools-6.6.4656-cp35-cp35m-macosx_10_6_intel.whl (7.4MB)

I'm not sure where the problem lies or how I should proceed. Any advice would be greatly appreciated!

[Mizux]

Hi,

Is it possible for you to use the last ortools package ?
cf https://pypi.python.org/pypi/ortools/6.7.4957 (should be built with protobuf 3.5.1)

side note: 6.6.4656 was also broken on mac (wrong loader path for pywraplp if I remember well) so we release https://pypi.python.org/pypi/ortools/6.6.4659 for MacOS...

As a last resort, rebuild OR-Tools locally and create your own wheel package...
(we also have a make install_python rule now 😃 )

[ghost]

Wow, that was quick! Thanks for responding!

I tried 6.7.4957 by just doing pip install ortools. Checked version:

> pip show ortools
Name: ortools
Version: 6.7.4957
Summary: Google OR-Tools python libraries and modules
Home-page: https://developers.google.com/optimization/
Author: Google Inc
Author-email: lperron@google.com
License: Apache 2.0
Location: /Users/nikolay/.pyenv/versions/3.5.4/envs/main/lib/python3.5/site-packages
Requires: protobuf, six

Unfortunately, 6.7.4957 still doesn't work - same thing happens as above!

What did work for me is building ortools with cmake, as detailed here. So my workaround is:

> git clone https://github.com/google/or-tools.git
> cd or-tools
> cmake -H. -Bbuild -G "Unix Makefiles"
> cmake --build build
> cd build
> pip install -e .

Would be a lot nicer to just use pip though.

[Mizux]

To build packages I used the script https://github.com/google/or-tools/blob/master/tools/build_delivery_unix.sh
so Makefile based build...

Protobuf is downloaded in dependencies/sources/protobuf-3.5.0 according to Dependencies.txt file.

then after you have some:
makefiles/Makefile.python.mk:195: $(PROTOBUF_DIR)/bin/protoc .....

So you can debug with make print-PROTOBUF_DIR

Is it possible that you have an old protobuf (3.0.0) version in your path/DYLD_LIBRARY_PATH ?
also check which protoc in doubt or env PROTOBUF_DIR.

note: CMake port is still experimental, happy you manage to build on MacOS with it...

[mmcloughlin]

Same problem. I suspect the problem is this amusing line

or-tools/makefiles/Makefile.third_party.unix.mk

Line 223 in 6961e30

git clone --quiet https://github.com/google/protobuf.git dependencies/sources/protobuf-$(PROTOBUF_TAG) && cd dependencies/sources/protobuf-$(PROTOBUF_TAG) && git checkout 3d9d1a1

The version is pinned to 3d9d1a1 not $(PROTOBUF_TAG) as you might think.

[Mizux]

@mmcloughlin Wow good catch !

patch in progress then I'll have to rebuild python package I guess...

[mmcloughlin]

Affects c++ too FYI.

[Mizux]

FYI Release 6.7.4973 aka tag v6.7.1 in progress...

[TilmanHS]

The dependency problem with the stringent version of Protobuf still persists
Doing a fresh install on Mac using

pipenv install ortools' gives:

.....
Successfully installed ortools-6.5.4527 protobuf-3.3.0 six-1.11.0
....
Installing dependencies from Pipfile.lock (9ef85d)...
An error occurred while installing ortools==6.7.4973! Will try again.
...
Could not find a version that satisfies the requirement ortools==6.7.4973 (from -r /var/folders/65/dwcd761n7_7g19d8yy6m17ph0000gn/T/pipenv-7syu7eec-requirements/pipenv-kf_s3ni9-requirement.txt (line 1)) (from versions: 6.5.4527)
No matching distribution found for ortools==6.7.4973 (from -r /var/folders/65/dwcd761n7_7g19d8yy6m17ph0000gn/T/pipenv-7syu7eec-requirements/pipenv-kf_s3ni9-requirement.txt (line 1))

Note the various different versions of ortools that are being mentioned.
The resulting installation contains a mismatch with protobuf:

pipenv graph

ortools==6.5.4527

• protobuf [required: ==3.3.0, installed: 3.6.0]
  • setuptools [required: Any, installed: 40.0.0]
  • six [required: >=1.9, installed: 1.11.0]

Any suggestions ?

[Mizux]

I think your problem is your MacOS version we are currently provided this macos pypi package
https://pypi.org/project/ortools/#files

Which version of python did you use ? which version of MacOS ?

[TilmanHS]

Mac: OSX El Capitan (10.11.6 (15G1510)
Python 3.6.4

[Mizux]

Just to confirm are you on a 64bits distro ?

what is the return of:

python3 --version
python3 -c "import platform; print(platform.architecture()[0])"
python3 -m pip --version

[TilmanHS]

Yes, 64 bits:

python version: Python 3.6.4
architecture : 64bit
pip version : pip 10.0.1 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip (python 3.6)

I went back to or-tools version 6.5.4527 to get things running again.

[tgsmith61591]

FWIW, I still couldn't get this to work with 6.7.4973. I had to revert to a lower version of protobuf:

$ pip install protobuf==3.5.1
Installing collected packages: protobuf
  Found existing installation: protobuf 3.6.1
    Uninstalling protobuf-3.6.1:
      Successfully uninstalled protobuf-3.6.1
Successfully installed protobuf-3.5.1

[Mizux]

v6.7 was based on protobuf 3.5.1 while started form v6.9 we now rely on protobuf 3.6
see: https://github.com/google/or-tools/releases/tag/v6.9

[ShoufaChen]

I met a similar error using torch SummaryWriter:

[libprotobuf FATAL google/protobuf/stubs/common.cc:83] This program was compiled against version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.20.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'

Solved by installing the required version directly using pip:

pip install protobuf==3.9.2