[Bug] Deserialization of exception fails

[Tehada]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

I have this code in example.py:

import urllib.error
import ray

ray.init(address="auto")


@ray.remote
def f():
    raise urllib.error.HTTPError("lol", code=1, msg="lol", hdrs=None, fp=None)


res = [f.remote()]
ray.get(res)

When I run this code inside latest docker rayproject/ray with default ray 1.8.0 I am getting this error:

(base) ray@comet:/app$ ray start --head
...
here successfully starting ray runtime
...
(base) ray@comet:/app$ python example.py
2021-12-14 06:25:26,599 INFO worker.py:823 -- Connecting to existing Ray cluster at address: 192.168.0.170:6379
2021-12-14 06:25:27,441 ERROR serialization.py:270 -- Failed to unpickle serialized exception
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 32, in from_bytes
    return pickle.loads(ray_exception.serialized_exception)
TypeError: __init__() missing 5 required positional arguments: 'url', 'code', 'msg', 'hdrs', and 'fp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 268, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 214, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 35, in from_bytes
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
Traceback (most recent call last):
  File "example.py", line 13, in <module>
    ray.get(res)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1627, in get
    raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 32, in from_bytes
    return pickle.loads(ray_exception.serialized_exception)
TypeError: __init__() missing 5 required positional arguments: 'url', 'code', 'msg', 'hdrs', and 'fp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 268, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 214, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 35, in from_bytes
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

(base) ray@comet:/app$

It seems that during deserialization of exception the constructor of urllib.error.HTTPError is called, which fails deserialization. Currenlty I am not sure that I have enough understanding about deserialization process to submit a PR.

Versions / Dependencies

running in default docker image rayproject/ray:latest with IMAGE ID 9c9d027f57e8.

Reproduction script

Described above.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Tehada]

Upd: seems that problem is in cloudpickle.py which resides in python/ray/cloudpickle:

In [1]: import urllib.error

In [2]: e = urllib.error.HTTPError("lol", code=1, msg="lol", hdrs=None, fp=None)

In [3]: import cloudpickle

In [4]: p = cloudpickle.dumps(e)

In [5]: cloudpickle.loads(p)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-dc63557f4f37> in <module>
----> 1 cloudpickle.loads(p)

TypeError: __init__() missing 5 required positional arguments: 'url', 'code', 'msg', 'hdrs', and 'fp'

In [6]:

[Tehada]

Upd2: it appears that urllib.error.HTTPError doesn't conform the definition of picklable class. I am not sure why python core devs designed it this way, but nonetheless, it would be cool to have some fallback solution in ray for exceptions which are not picklable.

[clarkzinzow]

Yep this is a common issue with urllib.error.HTTPError, which for some reason the Python devs made unpicklable. Another Ray issue was opened for this recently. However, given that we can't know that the exception isn't picklable until deserialization time, there's not much that Ray can do here other than what it's currently doing: indicating that deserializing an exception has failed, and dumping the traceback.

Closing this since this is fundamentally a urllib issue.