[Packaging][Python] macOS wheels broken: libprotobuf.18.dylib missing

[asfimport]

If I build macOS arrow wheels with crossbow from the latest master (a77257f) and install them, importing pyarrow gives the following error message:

In [1]: import pyarrow                                                                                                                                                               

---------------------------------------------------------------------------

ImportError                               Traceback (most recent call last)

<ipython-input-1-f1048abcb32d> in <module>

----> 1 import pyarrow




~/anaconda3/lib/python3.6/site-packages/pyarrow/__init__.py in <module>

     47 import pyarrow.compat as compat

     48

---> 49 from pyarrow.lib import cpu_count, set_cpu_count

     50 from pyarrow.lib import (null, bool_,

     51                          int8, int16, int32, int64,




ImportError: dlopen(/Users/pcmoritz/anaconda3/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/protobuf/lib/libprotobuf.18.dylib

  Referenced from: /Users/pcmoritz/anaconda3/lib/python3.6/site-packages/pyarrow/libarrow.14.dylib

  Reason: image not found

 

Reporter: Philipp Moritz / @pcmoritz
Assignee: Philipp Moritz / @pcmoritz

PRs and other links:

• GitHub Pull Request #4685

_{Note: This issue was originally created as ARROW-5690. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Two options:

• Set -DARROW_PROTOBUF_USE_SHARED=OFF or

• Bundle libprotobuf (it's being bundled for Windows already: https://github.com/apache/arrow/blob/master/python/CMakeLists.txt#L508)

  I'd vote for the 1st option for now. Can you submit a patch and test it?

[asfimport]

Wes McKinney / @wesm:
Another issue is why this is not properly tested in the wheel build itself

[asfimport]

Philipp Moritz / @pcmoritz:
Linking protobuf statically leads to the following error:

ImportError: dlopen(/Users/pcmoritz/anaconda3/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/grpc/lib/libgrpc++.dylib

  Referenced from: /Users/pcmoritz/anaconda3/lib/python3.6/site-packages/pyarrow/libarrow_python.14.dylib

  Reason: image not found

So we might need to bundle GRPC (but I'm not sure about that). Do we have any configurations in the build system where we do that at the moment?

[asfimport]

Wes McKinney / @wesm:
We should be statically linking gRPC already, so that's the core issue here. I suppose this must have to do with the contents of the "grpc" Homebrew package, we may have to change the package build to build gRPC from source if the static libraries are not found

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4685
#4685