Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sort with limit panics for the limit includes some but not all nulls, for large arrays #235

Closed
mathiaspeters-sig opened this issue Apr 28, 2021 · 0 comments · Fixed by #236
Closed

Sort with limit panics for the limit includes some but not all nulls, for large arrays #235

mathiaspeters-sig opened this issue Apr 28, 2021 · 0 comments · Fixed by #236
Labels
arrow Changes to the arrow crate bug

Comments

@mathiaspeters-sig
Copy link
Contributor

Describe the bug
We are getting the following panic:
thread panicked at 'source slice length (139885) does not match destination slice length (239865)' at https://github.com/apache/arrow-rs/blob/master/arrow/src/compute/kernels/sort.rs#L516
when we are trying to sort a large array with a large limit that includes some but not all of the nulls. We are on commit 0d1697f619c1bff8f64b19df5d4ec66634eaecf9.

To Reproduce
We have tried to reproduce this with smaller datasets but have been unable to. The smallest dataset we have confirmed this to happen on had 1mil rows.

Expected behavior
Sorting should work normally without panicing.

Additional context
We plan to work on this tomorrow (CET).
@jorgecarleitao jorgecarleitao added the arrow Changes to the arrow crate label Apr 28, 2021
medwards pushed a commit to medwards/arrow-rs that referenced this issue Apr 28, 2021
sort_primitive result is capped to the min of limit or values.len
fbd818a 
fixes apache#235
@medwards medwards mentioned this issue Apr 28, 2021
Merged
medwards added a commit to medwards/arrow-rs that referenced this issue Apr 28, 2021
@medwards
sort_primitive result is capped to the min of limit or values.len
f33a340 
fixes apache#235
medwards added a commit to medwards/arrow-rs that referenced this issue Apr 29, 2021
@medwards
sort_primitive result is capped to the min of limit or values.len
1ca8b45 
fixes apache#235
medwards added a commit to medwards/arrow-rs that referenced this issue Apr 29, 2021
@medwards
sort_primitive result is capped to the min of limit or values.len
17bcde9 
fixes apache#235
@alamb alamb closed this as completed in #236 May 1, 2021
alamb pushed a commit that referenced this issue May 1, 2021
@medwards
sort_primitive result is capped to the min of limit or values.len (#236)
4dfbca6 
* sort_primitive result is capped to the min of limit or values.len

fixes #235

* Fixed length calculation of nulls to include

* Add more sort_primitive tests for sorts /w limit
@jorgecarleitao jorgecarleitao mentioned this issue May 14, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sort_primitive result is capped to the min of limit or values.len medwards/arrow-rs
2 participants
@jorgecarleitao @mathiaspeters-sig