You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sometimes (specifically on inputs) it makes a lot of sense to enable stateful processing without additional shuffle. This is most valuable on inputs, because after the input, partitioning is no longer defined or guaranteed. We will enhance FlatMap operator, that will take a StatefulUnaryFunctor, which will be sort of rich function with enhanced life-cycle methods (setup(Context), cleanup(Context)) and the apply method will take StatefulContext, with access to StorageProvider. We will consider possibility to add this special version of flatmap operator only attached to inputs. This is optional, though.
The text was updated successfully, but these errors were encountered:
Sometimes (specifically on inputs) it makes a lot of sense to enable stateful processing without additional shuffle. This is most valuable on inputs, because after the input, partitioning is no longer defined or guaranteed. We will enhance
FlatMap
operator, that will take aStatefulUnaryFunctor
, which will be sort of rich function with enhanced life-cycle methods (setup(Context)
,cleanup(Context)
) and theapply
method will takeStatefulContext
, with access toStorageProvider
. We will consider possibility to add this special version of flatmap operator only attached to inputs. This is optional, though.The text was updated successfully, but these errors were encountered: