Skip to content

fix(ray): restore ray client functionality [ENG-339]#119

Merged
eywalker merged 2 commits intonauticalab:devfrom
brian-arnold:re-restore-ray-executor
Mar 28, 2026
Merged

fix(ray): restore ray client functionality [ENG-339]#119
eywalker merged 2 commits intonauticalab:devfrom
brian-arnold:re-restore-ray-executor

Conversation

@brian-arnold
Copy link
Copy Markdown
Collaborator

Fix: Restore asyncio.wrap_future() in RayExecutor.async_execute_callable()

Problem

RayExecutor.async_execute_callable() fails with TypeError: object Future can't be used in 'await' expression when connected to a Ray cluster via the
client protocol (ray:// addresses).

In Ray client mode, ClientObjectRef.future() returns a concurrent.futures.Future, not an asyncio.Future. Directly awaiting it raises a TypeError,
causing all packets to fail when using the AsyncPipelineOrchestrator with a remote Ray cluster.

Root Cause

Commit c1fe4bd (March 26, "fix(review): address Copilot PR review comments") removed asyncio.wrap_future() from async_execute_callable(), replacing:

raw, stdout_log, stderr_log, python_logs = await asyncio.wrap_future(ref.future())

with:

raw, stdout_log, stderr_log, python_logs = await ref.future()

The commit message stated "Ray's ObjectRef.future() already returns an asyncio.Future, making wrap_future unnecessary." This is true for local Ray
(ObjectRef), but not for Ray client mode (ClientObjectRef), which returns concurrent.futures.Future. The existing test was also modified to assert
the broken behavior.

Fix

  • Restored asyncio.wrap_future(ref.future()) in async_execute_callable(). This is safe for both local and client modes — asyncio.wrap_future()
    correctly handles both Future types.
  • Updated the source-inspection regression test to assert wrap_future is present and document why it's required.
  • Added a behavioral test that mocks ref.future() returning a concurrent.futures.Future resolved from a background thread (mimicking Ray client
    mode) and verifies the executor correctly awaits it.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@brian-arnold brian-arnold changed the title fix(ray): restore ray client functionality fix(ray): restore ray client functionality [ENG-339] Mar 27, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Restores Ray client-mode compatibility for RayExecutor.async_execute_callable() by reintroducing asyncio.wrap_future() when awaiting Ray *.future() results, preventing TypeError when using ray:// addresses with async orchestration.

Changes:

  • Reinstates await asyncio.wrap_future(ref.future()) in RayExecutor.async_execute_callable().
  • Updates the regression source-inspection test to assert wrap_future usage and explains the Ray client-mode requirement.
  • Adds a behavioral regression test that simulates ClientObjectRef.future() returning a concurrent.futures.Future resolved from a background thread.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/orcapod/core/executors/ray.py Restores asyncio.wrap_future(ref.future()) to correctly await Ray client futures.
tests/test_core/test_regression_fixes.py Updates/extends regression coverage for Ray async execution in client mode.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 583 to 587
assert "wrap_future" in source, (
"async_execute_callable must use asyncio.wrap_future() for "
"Ray client mode compatibility — ClientObjectRef.future() "
"returns concurrent.futures.Future, not asyncio.Future"
)
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source-inspection assertion is too weak: it only checks that the substring "wrap_future" appears anywhere in the function source, which would still pass if the call to asyncio.wrap_future(...) were removed but the docstring/comment remained. Consider asserting a more specific pattern (e.g., "await asyncio.wrap_future(" and/or "asyncio.wrap_future(ref.future()"), and optionally asserting that there is no bare await ref.future() call.

Suggested change
assert "wrap_future" in source, (
"async_execute_callable must use asyncio.wrap_future() for "
"Ray client mode compatibility — ClientObjectRef.future() "
"returns concurrent.futures.Future, not asyncio.Future"
)
# Ensure we actually await asyncio.wrap_future(...)
assert "await asyncio.wrap_future(" in source, (
"async_execute_callable must await asyncio.wrap_future(...) for "
"Ray client mode compatibility — ClientObjectRef.future() "
"returns concurrent.futures.Future, not asyncio.Future"
)
# Ensure the wrapped object is ref.future(), not something else
assert "asyncio.wrap_future(ref.future()" in source, (
"async_execute_callable must call asyncio.wrap_future(ref.future()) "
"to correctly bridge concurrent.futures.Future returned by "
"ClientObjectRef.future() into the asyncio world"
)
# Guard against the incorrect pattern of awaiting ref.future() directly
assert "await ref.future(" not in source, (
"async_execute_callable must not await ref.future() directly; "
"it must use await asyncio.wrap_future(ref.future()) instead"
)

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted, thanks — tightened the assertions to check for the specific await asyncio.wrap_future( pattern and guard against bare await ref.future().

Comment on lines +213 to +218
# In Ray client mode (ray://) ClientObjectRef.future() returns a
# concurrent.futures.Future which cannot be directly awaited.
# asyncio.wrap_future() handles both Future types correctly.
raw, stdout_log, stderr_log, python_logs = await asyncio.wrap_future(
ref.future()
)
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says asyncio.wrap_future() "handles both Future types"; in stdlib asyncio, wrap_future() only accepts concurrent.futures.Future (passing an asyncio.Future raises TypeError). Either tighten the comment to state that Ray's *.future() returns a concurrent.futures.Future (in the modes you support), or add a small type check so async_execute_callable can correctly await either an asyncio.Future or a concurrent.futures.Future.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is actually correct — asyncio.wrap_future() does handle both types. Per the Python docs: "If the argument is already an asyncio.Future, it is returned unchanged." So no type check is needed — wrap_future is safe for both local Ray (asyncio.Future) and client mode (concurrent.futures.Future).

@brian-arnold brian-arnold requested a review from eywalker March 27, 2026 22:31
@brian-arnold brian-arnold marked this pull request as draft March 27, 2026 23:03
@brian-arnold brian-arnold marked this pull request as ready for review March 28, 2026 00:08
Copy link
Copy Markdown
Contributor

@eywalker eywalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the regression I've introduced!

@eywalker eywalker merged commit 01b0730 into nauticalab:dev Mar 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants