Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][experimental] Support tasks with multiple return values in accelerated DAG #45569

Closed
stephanie-wang opened this issue May 25, 2024 · 1 comment · Fixed by #47024
Closed
Assignees
Labels
compiled-graph core Issues that should be addressed in Ray Core enhancement Request for new feature and/or capability P0 Issues that should be fixed in short order

Comments

@stephanie-wang
Copy link
Contributor

Description

Accelerated DAGs currently error if a task with multiple return values is found.

In [11]: @ray.remote
    ...: class Actor:
    ...:     @ray.method(num_returns=2)
    ...:     def forward(self, x):
    ...:         return x, x + 1
    ...: 

In [12]: a = Actor.remote()

In [13]: with ray.dag.InputNode() as inp:
    ...:     dag = a.forward.bind(inp)
    ...: 

In [14]: dag.execute(0)
Out[14]: 
[ObjectRef(32d950ec0ccf9d2ad9e7e77e45dee424649f688f0100000001000000),
 ObjectRef(32d950ec0ccf9d2ad9e7e77e45dee424649f688f0100000002000000)]

In [15]: ray.get(dag.execute(0))
Out[15]: [0, 1]

In [16]: dag = dag.experimental_compile()

In [17]: ray.get(dag.execute(0))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 1
----> 1 ray.get(dag.execute(0))

File ~/ray/python/ray/_private/auto_init_hook.py:21, in wrap_auto_init.<locals>.auto_init_wrapper(*args, **kwargs)
     18 @wraps(fn)
     19 def auto_init_wrapper(*args, **kwargs):
     20     auto_init_ray()
---> 21     return fn(*args, **kwargs)

File ~/ray/python/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    101     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    102         return getattr(ray, func.__name__)(*args, **kwargs)
--> 103 return func(*args, **kwargs)

File ~/ray/python/ray/_private/worker.py:2617, in get(object_refs, timeout)
   2614     object_refs = [object_refs]
   2616 if not isinstance(object_refs, list):
-> 2617     raise ValueError(
   2618         f"Invalid type of object refs, {type(object_refs)}, is given. "
   2619         "'object_refs' must either be an ObjectRef or a list of ObjectRefs. "
   2620     )
   2622 # TODO(ujvl): Consider how to allow user to retrieve the ready objects.
   2623 values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)

ValueError: Invalid type of object refs, <class 'ray.experimental.channel.SynchronousReader'>, is given. 'object_refs' must either be an ObjectRef or a list of ObjectRefs. 

Use case

No response

@stephanie-wang stephanie-wang added enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks compiled-graph labels May 25, 2024
@rkooo567
Copy link
Contributor

rkooo567 commented Aug 8, 2024

we can work around it with tuple for now I think. Let's revisit it after finishing existing P1s?

@rkooo567 rkooo567 added P2 Important issue, but not time-critical and removed P1 Issue that should be fixed within a few weeks labels Aug 8, 2024
@rkooo567 rkooo567 added core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order and removed P2 Important issue, but not time-critical labels Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiled-graph core Issues that should be addressed in Ray Core enhancement Request for new feature and/or capability P0 Issues that should be fixed in short order
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants