-
Notifications
You must be signed in to change notification settings - Fork 662
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(data): map type ID to child index before indexing a union child array #4598
fix(data): map type ID to child index before indexing a union child array #4598
Conversation
abb5612
to
00edcba
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, looks good to me. Just minor suggestion that you can feel free to ignore
@@ -48,14 +51,18 @@ pub(super) fn build_extend_dense(array: &ArrayData) -> Extend { | |||
.extend_from_slice(&type_ids[start..start + len]); | |||
|
|||
(start..start + len).for_each(|i| { | |||
let type_id = type_ids[i] as usize; | |||
let type_id = type_ids[i]; | |||
let child_index = src_fields |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we could follow the approach used by UnionArray and compute this mapping once
https://github.com/apache/arrow-rs/blob/master/arrow-array/src/array/union_array.rs#L345
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually I implemented that optimization in an earlier version, but then realized that it might be counterproductive and excessively memory-intensive if the number of rows per input array is very low, and the number of input arrays is large (as in my use case). equal_dense
is implemented in the naïve way as well.
00edcba
to
ab724b1
Compare
Force-pushed a minor change to sate clippy |
Which issue does this PR close?
Closes #4578.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?