Generalize ReduceDelegate #120
Conversation
fpacifici
force-pushed
the
fpacifici/actually_port_reduce
branch
2 times, most recently
from
b0c89f5 to
be45209
Compare
fpacifici
force-pushed
the
fpacifici/support_parallelism
branch
from
0e26379 to
b10c52e
Compare
| - The Arroyo Reduce strategy is an Arroyo strategy, so it needs to be | ||
| fed with Arroyo messages, thus the adaptation logic from the | ||
| new Streaming platform message that the Rust code deals with. |
There was a problem hiding this comment.
Is the idea here, to make it simpler to adapt existing consumers built as arroyo tasks into the streaming platform? That seems like a great way to make the conversion process simpler.
There was a problem hiding this comment.
Yes, it can be used for that as well.
Here the goal was to adapt Arroyo strategies, but a consumer in the end is a chain of strategies so that works as well.
| TStrategyOut = TypeVar("TStrategyOut") | ||
|
|
||
|
|
||
| class OutputRetriever(ProcessingStrategy[TStrategyOut], Generic[TStrategyOut]): |
There was a problem hiding this comment.
In a nutshell, how is this OutputRetriever different from the old one?
| make them available to rust. | ||
| The message flow looks like this: | ||
| 1. The Rust Arroyo Strategy receives a streaming message to process. |
There was a problem hiding this comment.
Thanks for outlining the steps here. It is much clearer now
ayirr7
left a comment
ayirr7
left a comment
There was a problem hiding this comment.
This overall looks good to me. I'll wait for any others' who might have feedback before I approve
Based on #119
While buildign a RustOperatorDelegate for the RunTaskInMultiProcess
I noticed that the delegating logic was very similar to the Reduce
step so I make the
OutputRetrieverand theReduceDelegatereusable.This PR:
functions passed to the two classes
Message, but that is a type Rust cannot understand.