[Bug] Can't pickle function objects error on ray.init

[btseytlin]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

Running ray.init() throws this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 950, in init
    job_config=job_config)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 1451, in connect
    lambda worker_info: sys.path.insert(1, current_directory))
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 386, in run_function_on_all_workers
    pickled_function = pickle.dumps(function)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 485, in dump
    self.save(obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 601, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 689, in save_reduce
    save(func)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 601, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 689, in save_reduce
    save(func)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 576, in save
    rv = reduce(self.proto)
TypeError: can't pickle function objects
>>> 

Versions / Dependencies

Ubuntu 20.04, Python 3.7.1 via pyenv, ray 1.7.1

Reproduction script

import ray
ray.init()

Anything else

Original discussion in Modin project: modin-project/modin#3601

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rkooo567]

Is it reproducible from conda env? (doesn't mean you should use the conda env, but I'd like to understand if it is the pyenv issue)

[btseytlin]

Is it reproducible from conda env? (doesn't mean you should use the conda env, but I'd like to understand if it is the pyenv issue)

I have no idea, I don't use conda

[MXD6]

I encountered the same problem, how did you solve it？

[btseytlin]

I encountered the same problem, how did you solve it？

I did not! Still looking for a solution and hoping Ray team will help

[MXD6]

This issue has been solved for Python 3.8.2rc1, Python 3.9.0 alpha 4 or late versions.

[rkooo567]

@MXD6 Do you mind trying download pip install pickle5 dependency for python 3.7, and see if it works? (This one shouldn't be installed from python 3.8, but needs to be downloaded for the lower python I believe)

[btseytlin]

This issue has been solved for Python 3.8.2rc1, Python 3.9.0 alpha 4 or late versions.

My issue is that I need it to work on python 3.7 :(

[btseytlin]

@MXD6 Do you mind trying download pip install pickle5 dependency for python 3.7, and see if it works? (This one shouldn't be installed from python 3.8, but needs to be downloaded for the lower python I believe)

I will try it as soon as I can

[Ivanyan0516]

have you solved this problem?

[btseytlin]

@MXD6 Do you mind trying download pip install pickle5 dependency for python 3.7, and see if it works? (This one shouldn't be installed from python 3.8, but needs to be downloaded for the lower python I believe)

It did not resolve the issue:

boris@boris-Legion-Y540-15IRH:~/Documents/mindsdb/dfsql$ pip install pickle5
Collecting pickle5
  Using cached pickle5-0.0.11.tar.gz (132 kB)
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for pickle5, since package 'wheel' is not installed.
Installing collected packages: pickle5
    Running setup.py install for pickle5 ... done
Successfully installed pickle5-0.0.11
boris@boris-Legion-Y540-15IRH:~/Documents/mindsdb/dfsql$ python
Python 3.7.1 (default, Oct 26 2021, 13:26:12) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
ray.>>> ray.init()
2021-11-08 15:21:42,726 INFO services.py:1255 -- View the Ray dashboard at http://127.0.0.1:8265
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 950, in init
    job_config=job_config)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 1451, in connect
    lambda worker_info: sys.path.insert(1, current_directory))
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/worker.py", line 386, in run_function_on_all_workers
    pickled_function = pickle.dumps(function)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 485, in dump
    self.save(obj)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 601, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 689, in save_reduce
    save(func)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 601, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 689, in save_reduce
    save(func)
  File "/home/boris/.pyenv/versions/3.7.1/lib/python3.7/site-packages/ray/pickle5_files/pickle5/pickle.py", line 576, in save
    rv = reduce(self.proto)
TypeError: can't pickle function objects
>>> 

[Sud0x67]

Is it reproducible from conda env? (doesn't mean you should use the conda env, but I'd like to understand if it is the pyenv issue)

I use conda env and encounter the same problem.

[AliKarimi95]

The same problem occurs with conda env and Python 3.7.1

[Ramtin-Nouri]

This seems to be an known issue with Tensorflow2.
I had the same problem as you. Even when I was not using Tensorflow and only doing ray.init() would show the same crash.
Creating a new virtual environment and installing tensorflow 1.4 instead fixed it for me.

[rkooo567]

It looks like there are many people going through the issue. Can you try setting up the priority cc @ericl ?

[rkooo567]

@Ramtin-Nouri are you seeing this error even when you didn't import tensorflow at all from the python script?

[Ramtin-Nouri]

@rkooo567 Yes. Even this would show the crash:

import ray
ray.init()

It seems as if only tensorflow 2 is installed its enough for the error to occur !?
Tensorflow 2 is installed on my system without virtual environment.
Thus creating a new empty virtual environment without system-site-packages fixed the problem.
Installing Tensorflow 1.4 to the venv also did not reintroduce the error.

[wumpus]

I ran into this problem after I decided to use pickle5 to check that my users aren't calling ray with too large of an argument list.

My environment is ray 1.2.0, pickle5 0.0.12

Whether my code includes pickle5 or not, ray.init() fails.

When I "pip uninstall pickle5", all is well.

I don't have any tensorflow installed in this venv, so this pickle5 problem appears to be the fundamental problem. Presumably ray has a conditional import for it.

[shanyu-sys]

Same problem with 3.7.0. I have no tensorflow on my system without virtual environment or pickle5
Problem disappeared with python==3.7.10.

It might be a issue related to python.

[wumpus]

You might want to examine your stack trace, it would be surprising if pickle5 is mentioned there if it's not installed.

The python version dependency might be that 3.7 backported some stuff in a middle version. The entire pickle5 module is unusual because it's a backport of the 3.8 pickle for earlier Python versions.

[MiksVasiljevs]

Same problem with 3.7.0. I have no tensorflow on my system without virtual environment or pickle5 Problem disappeared with python==3.7.10.

It might be a issue related to python.

Upgrading from 3.7.0 to 3.7.10 resolved this for me. Thanks!

[rkooo567]

For me, I had this issue when I built Ray from sources. It was resolved when I re-create the virtual env

[HansBug]

I found this too. Ubuntu 20.04, python3.6.3, ray1.12.1.

[tupui]

After reading this thread, it seems that it's not a problem with Ray and this issue could be closed.

[AishuaiYao]

This issue has been solved for Python 3.8.2rc1, Python 3.9.0 alpha 4 or late versions.

this method can solve my problem