You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi there, I can absolutely see the benefit of this crate for short lived jobs, and it looks really great for that. However, can I use this for long lived and stateful jobs, as the basis of an actor model?
For example, I have a bunch (~10) of different Processes which might "wake up" at certain times and push work onto a central queue. These processes are:
stateful
CPU intensive when they have work to do, which might itself be parallellellellisable
live for the life of the application
As an example, one of these Processes might sweep through the system every couple of hours, noticing and flagging any stale data for review.
(A simple std approach would have a ThreadPool with each Process being a thread using channels to chat to each other and the central queue.)
I can see how each Process could use rayon, and I can see how I could use rayon in an event loop calling the individual Processes which then die until the next loop, but for long lived stateful jobs? I'm struggling to see if rayon is a good fit there?
Am I over thinking this (just for outer Processes)? Should I just use rayon and loop/yield within the processes? Should I just use a normal thread pool and move onto more interesting things?
Thanks!
P.S. Google gave me no hits for rayon and actor, which was a big sign I might be missing some foot gun here.
The text was updated successfully, but these errors were encountered:
Should I just use rayon and loop/yield within the processes?
I think the footgun there is that rayon yielding is nested, blocking that code on the stack by whatever gets called next. This is not like async/await that captures all the state to move off the stack. So, you would not want to be in a situation where one long-running loop yields to another, because only the inner one will get to make progress.
You could just create a dedicated thread for each actor, so they can loop with independent progress, and then have them farm their CPU intensive work to the shared rayon pool as needed. You could also write the actors in a true async/await framework, which may come with its own threadpool, or again lean on rayon for the CPU work. For example, here's a blog post on that topic, Actors with Tokio.
Hi there, I can absolutely see the benefit of this crate for short lived jobs, and it looks really great for that. However, can I use this for long lived and stateful jobs, as the basis of an actor model?
For example, I have a bunch (~10) of different Processes which might "wake up" at certain times and push work onto a central queue. These processes are:
As an example, one of these Processes might sweep through the system every couple of hours, noticing and flagging any stale data for review.
(A simple
std
approach would have a ThreadPool with each Process being a thread using channels to chat to each other and the central queue.)I can see how each Process could use
rayon
, and I can see how I could userayon
in an event loop calling the individual Processes which then die until the next loop, but for long lived stateful jobs? I'm struggling to see if rayon is a good fit there?Am I over thinking this (just for outer Processes)? Should I just use
rayon
and loop/yield within the processes? Should I just use a normal thread pool and move onto more interesting things?Thanks!
P.S. Google gave me no hits for
rayon
andactor
, which was a big sign I might be missing some foot gun here.The text was updated successfully, but these errors were encountered: