You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Operators that maintain state can be migrated to use the pipeline state mechanism (eg: for the batch or backpressure operator) -- this would be a necessity when we want to support pipeline migration to a different tremor node, when we have clustering for tremor.
Describe the solution you'd like
Use the pipeline state mechanism for these operators:
generic::batch
generic::backpressure
Come up with a solution for these operators, that have internal state not easily expressible as Value (our primitive for the operator state). One solution is to serialize the stateful data structures into (and deserialize out) of the pipeline state, when we do need to store them somewhere central (eg: to enable pipeline migrations as part of clustering effort).
Describe the problem you are trying to solve
Operators that maintain state can be migrated to use the pipeline state mechanism (eg: for the batch or backpressure operator) -- this would be a necessity when we want to support pipeline migration to a different tremor node, when we have clustering for tremor.
Describe the solution you'd like
Use the pipeline state mechanism for these operators:
Come up with a solution for these operators, that have internal state not easily expressible as
Value
(our primitive for the operator state). One solution is to serialize the stateful data structures into (and deserialize out) of the pipeline state, when we do need to store them somewhere central (eg: to enable pipeline migrations as part of clustering effort).Notes
Pulled from https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#future-possibilities to track as an issue.
Also see https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#unresolved-questions
The text was updated successfully, but these errors were encountered: