You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After removing explicit partitioning, we have currently no explicit control over the parallelism of executing operators. This affects both batch and stream. There must be a way to give a hint to the translator that certain operation should be parallelized more or less than the input. Options are:
add a method to set parallelism of operator on executor - e.g.
I think we should never set explicit parallelism, instead we should hint operator with the percentual estimate of increase / decrease in data size, so we can decide parallelism based on the input data.
After removing explicit partitioning, we have currently no explicit control over the parallelism of executing operators. This affects both batch and stream. There must be a way to give a hint to the translator that certain operation should be parallelized more or less than the input. Options are:
The text was updated successfully, but these errors were encountered: