[Bug] Ray 1.7 and 1.8 ray.init hangs when some versions of Python gRPC library is used due to global symbol export

[kiralaz]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

This is not a contribution.

After upgrading to Ray 1.7, we noticed that when we import pybinded C++ code that uses a different version of grpc than comes with ray, we consistently get segfaults.

Turns out this is due to Ray exporting its grpc symbols so they are global and available to things like _streaming.so. This seems to have been added in #18490

When I comment out *grpc*; at

ray/src/ray/ray_version_script.lds

Line 43 in 2367a2c

*grpc*;

(so the symbols are not exported) and build ray 1.7.0 locally, our code does not segfault.

Looking at #18870, it seems like *absl*; caused a similar issue and has already been removed, but @ashione mentions in #18870 (comment) that *grpc*; might be needed for some ray/streaming tests to pass.

Would it be possible for someone to look into this and remove grpc; ? I think it would be preferable to not export global symbols for commonly used libs like grpc and absl, to avoid causing issues for anyone using ray alongside C extensions built with other versions of these libs.

This is currently blocking my team from upgrading to ray 1.7+.

Thanks!

Versions / Dependencies

ray==1.7.0

Reproduction script

N/A

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rkooo567]

Cc @scv119 I think we should handle this (and probably fine streaming tests not passing)

[ashione]

@rkooo567 This #20136 might fix symbol-wise issuse. I put duplicated symbols into _streaming.so.

[rkooo567]

Oh nice. Let’s merge it and see if it fixes the issue

[fishbone]

@ashione assign this issue to you since you are working on this.

[rkooo567]

Hey @kiralaz can you check the latest master if this is resolved? We merged a PR that could solve the issue

[kiralaz]

Hey @kiralaz can you check the latest master if this is resolved? We merged a PR that could solve the issue

Thanks @rkooo567! #20136 looks like it should fix my issue.

Though when I tried python3 setup.py install from ray/python on master, I got

In file included from src/ray/util/filesystem.cc:15:0:
bazel-out/k8-opt/bin/_virtual_includes/ray_util/ray/util/filesystem.h:17:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.

For the bazel build of some ray_pkg targets:

subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.

[rkooo567]

I think the build requires gcc >=9.0. Can you check ur compiler version?

[kiralaz]

I see, I was on gcc 7. It's a bit tricky for me to upgrade my gcc right now, but saw the PR landed 23 hours ago so I just tried via the nightly wheel from https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl assuming the fix made it to the latest build.

Happy to report I didn't get a segfault 🎉 😄

[rkooo567]

Glad to hear that! I will close the issue. @scv119 @ericl we should consider having a patch release with this commit. I've seen other users who went through the same issue. Also we should consider adding code owner to the src/ray/ray_version_script.lds file.

[rkooo567]

This is fixed in the master, and the change will be included in 1.9

[kiralaz]

Thank you @rkooo567 ! And @ashione for the fix 🙂

[rkooo567]

@edoakes Was there any special reason to unpin this btw? I think we should pin it until 1.9 is released! (It can happen quite commonly)

[edoakes]

@rkooo567 sorry I have no idea how I did that, that was accidental

[simon-mo]

@rkooo567 1.9 is release now, should we unpin?

[rkooo567]

Yes! Please go ahead.