Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite clustering using Ray Actors #32

Open
jdrusso opened this issue Oct 25, 2022 · 5 comments
Open

Rewrite clustering using Ray Actors #32

jdrusso opened this issue Oct 25, 2022 · 5 comments
Assignees
Labels
enhancement New feature or request

Comments

@jdrusso
Copy link
Owner

jdrusso commented Oct 25, 2022

Clustering in particular is a stateful operation, which currently relies on serializing/deserializing modelWE objects from the Ray object store.

Ideally, there isn't much overhead associated with this, but I think it becomes noticeable on systems without shared memory between workers.

Instead of doing the parallelism via Ray processes, we can initialize a set of Actors to do work. Actors are stateful, so we can just initialize them with the current model state (with unnecessary stuff stripped out).

@jdrusso jdrusso added the enhancement New feature or request label Oct 25, 2022
@jdrusso
Copy link
Owner Author

jdrusso commented Oct 25, 2022

Maybe this would work well with a broader rewrite of the parallelism as a work manager interface

@jdrusso jdrusso self-assigned this Nov 2, 2022
@jdrusso
Copy link
Owner Author

jdrusso commented Nov 2, 2022

@SHZ66 This is what we were talking about the other day, just tagging you here

@SHZ66
Copy link
Contributor

SHZ66 commented Nov 4, 2022

@jdrusso sounds good. I am gonna work on this and the next week or so. Feel free to assign the issue to me too

@jdrusso
Copy link
Owner Author

jdrusso commented Nov 4, 2022

Assigned you, thanks for working on this!

@jdrusso
Copy link
Owner Author

jdrusso commented Nov 4, 2022

@SHZ66 Maybe something to keep in mind as you're doing this: #37

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants