Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate stateful operators to use pipeline state mechanism #71

Open
anupdhml opened this issue Feb 26, 2020 · 1 comment
Open

Migrate stateful operators to use pipeline state mechanism #71

anupdhml opened this issue Feb 26, 2020 · 1 comment
Labels
enhancement New feature or request operator Pipeline Operators

Comments

@anupdhml
Copy link
Contributor

Describe the problem you are trying to solve

Operators that maintain state can be migrated to use the pipeline state mechanism (eg: for the batch or backpressure operator) -- this would be a necessity when we want to support pipeline migration to a different tremor node, when we have clustering for tremor.

Describe the solution you'd like

Use the pipeline state mechanism for these operators:

  • generic::batch
  • generic::backpressure

Come up with a solution for these operators, that have internal state not easily expressible as Value (our primitive for the operator state). One solution is to serialize the stateful data structures into (and deserialize out) of the pipeline state, when we do need to store them somewhere central (eg: to enable pipeline migrations as part of clustering effort).

  • grouper::buckert (maintains LRU cache)

Notes

Pulled from https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#future-possibilities to track as an issue.

Also see https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#unresolved-questions

@anupdhml anupdhml added the enhancement New feature or request label Feb 26, 2020
@anupdhml
Copy link
Contributor Author

For reference, generic::counter operator is already using the state mechanism:
https://github.com/wayfair-tremor/tremor-runtime/blob/v0.7.3/tremor-pipeline/src/op/generic/counter.rs

Implementation PR it was part of: #59

@anupdhml anupdhml added offramp Offramps and removed offramp Offramps labels Feb 26, 2020
@anupdhml anupdhml added the operator Pipeline Operators label Aug 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request operator Pipeline Operators
Projects
None yet
Development

No branches or pull requests

1 participant