Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error importing ray in colab due to pyarrow #5033

Open
philwinder opened this issue Jun 25, 2019 · 3 comments

Comments

@philwinder
Copy link

commented Jun 25, 2019

System information

  • colab
  • pip install ray[rllib]
  • python 3.7

Describe the problem

Very simple, just pip install ray, then import ray:

!pip install tensorflow  # or tensorflow-gpu
!pip install ray[rllib]  # also recommended: ray[debug]
import ray

on Google colab and it fails with:

ImportError: Ray must be imported before pyarrow because Ray requires a specific version of pyarrow (which is packaged along with Ray).

Recreate

Take a look at this gist running on colab

@pcmoritz

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Thanks for reporting this! This error happens because colab is preloading certain libraries (more on this below). To fix it, you can do the following:

!pip install tensorflow  # or tensorflow-gpu
!pip install ray[rllib]  # also recommended: ray[debug]
!pip uninstall -y pyarrow

and then click on Runtime/Restart runtime. After that, the Ray cell will work.

Now as to why colab's preloading is problematic: Ray depends on a library called pyarrow for serialization and some other things. Because TensorFlow's binary package is not compliant with the manylinux1 standard [1], we need to ship a custom version of pyarrow that is compatible with TensorFlow. Until TensorFlow is fixed, we don't really have a choice here, so sorry for that.

[1] https://groups.google.com/a/tensorflow.org/d/msg/developers/TMqRaT-H2bI/J1Wiu8_XCQAJ

@philwinder

This comment has been minimized.

Copy link
Author

commented Jun 25, 2019

Ah yes, you are right. I had tried the uninstall command, but I had not restarted the runtime after uninstalling. That is the trick. Thank you for your quick response.

Can I leave it to you to decide whether you want to close the issue? Because it is obviously an issue, but like you said it is out of your control (unless you can remove the pyarrow dependency maybe?).

@pcmoritz

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

Yeah, we can leave it open for now, since it actually is an issue and might make it easier for people to find. If this happens broadly on colab, we can add a pointer in the error message on what to do. Dropping pyarrow is not really an option unfortunately because our fast serialization depends on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.