Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stateful mapping #192

Closed
je-ik opened this issue Nov 23, 2017 · 1 comment
Closed

Add stateful mapping #192

je-ik opened this issue Nov 23, 2017 · 1 comment

Comments

@je-ik
Copy link
Contributor

je-ik commented Nov 23, 2017

Sometimes (specifically on inputs) it makes a lot of sense to enable stateful processing without additional shuffle. This is most valuable on inputs, because after the input, partitioning is no longer defined or guaranteed. We will enhance FlatMap operator, that will take a StatefulUnaryFunctor, which will be sort of rich function with enhanced life-cycle methods (setup(Context), cleanup(Context)) and the apply method will take StatefulContext, with access to StorageProvider. We will consider possibility to add this special version of flatmap operator only attached to inputs. This is optional, though.

@je-ik
Copy link
Contributor Author

je-ik commented Nov 14, 2018

Inputs are handled by beam IOs and we have found no useful application of stateful mapping.

@je-ik je-ik closed this as completed Nov 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant