Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update RayTaskRunner for compatibility with new engine #13575

Merged
merged 9 commits into from
May 28, 2024
Merged

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented May 26, 2024

Updates the RayTaskRunner to be compatible with the new engine. I also moved the .map logic on the TaskRunner subclasses to the base class since it was the same between all three TaskRunner implementations.

I wanted to create a lower-level utility for running tasks with Ray's @ray.remote decorator, but Ray only allows functions and classes to be decorated with @ray.remote,, and I was having trouble smuggling a task_run_id on the returned ObjectRef. I think there's still lower-level utility hiding in there, but I figured I'd work backward from a working task runner.

Example

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

  • This pull request includes redirect settings in netlify.toml for files that are removed or renamed.

For new functions or classes in the Python SDK:

  • This pull request includes helpful docstrings.
  • If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in mkdocs.yml navigation.

@@ -11,7 +11,7 @@
from prefect.utilities.engine import collect_task_run_inputs_sync


class PrefectDistributedClient(Client):
class PrefectDaskClient(Client):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also renamed this client to be more consistent with the DaskTaskRunner and PrefectDaskFuture.

@desertaxle desertaxle marked this pull request as ready for review May 27, 2024 20:32
@desertaxle desertaxle requested review from zzstoatzz and a team as code owners May 27, 2024 20:32
@desertaxle desertaxle merged commit e838206 into main May 28, 2024
33 checks passed
@desertaxle desertaxle deleted the ray-wrapper branch May 28, 2024 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants