-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Data][Split] stable version of split with hints #26778
Conversation
f6120b2
to
fbf1992
Compare
BlockRefWithMeta = Tuple[ObjectRef[Block], BlockMetadata] | ||
|
||
|
||
def _equalize( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we add unit tests for these?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The overall algorithm seems sensible. It would be great to also pull out the greedy split algorithm into a helper function / unit test it, but that can be left for another PR.
a8c3270
to
e923f1f
Compare
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: nanqi.yxf <nanqi.yxf@antgroup.com>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: klwuibm <kwu888@gmail.com>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: Catch-Bull <burglarralgrub@gmail.com>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: Rohan138 <rapotdar@purdue.edu>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: Frank Luan <lsf@berkeley.edu>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: Scott Graham <scgraham@microsoft.com>
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks.
Why are these changes needed? Introduce a stable version of split with hints with a stable equalizing algorithm: use the greedy algorithm to generate the initial unbalanced splits. for each splits, first shave them so the number for rows are below the target_size based on how many rows needed for each split, do a one time split_at_index to the left over blocks. merge the shaved splits with the leftover splits. The guarantee of this algorithm is we at most need to split O(split) number of blocks. Signed-off-by: Stefan van der Kleij <s.vanderkleij@viroteq.com>
Signed-off-by: scv119 scv119@gmail.com
Why are these changes needed?
Introduce a stable version of
split with hints
with a stable equalizing algorithm:split_at_index
to the left over blocks.The guarantee of this algorithm is we at most need to split O(split) number of blocks.
Related issue number
Checks
scripts/format.sh
to lint the changes in this PR.