-
Notifications
You must be signed in to change notification settings - Fork 721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add type_ids in Union datatype #1703
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1703 +/- ##
==========================================
+ Coverage 83.25% 83.32% +0.07%
==========================================
Files 195 195
Lines 55906 56012 +106
==========================================
+ Hits 46544 46672 +128
+ Misses 9362 9340 -22
Continue to review full report at Codecov.
|
f2dfab6
to
29d0d16
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @viirya
Would it be correct to say that this PR also "Adds IPC support for UnionArray with non increasing integer type ids" ?
arrow/src/datatypes/datatype.rs
Outdated
@@ -115,7 +115,7 @@ pub enum DataType { | |||
/// A nested datatype that contains a number of sub-fields. | |||
Struct(Vec<Field>), | |||
/// A nested datatype that can represent slots of differing types. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// A nested datatype that can represent slots of differing types. | |
/// A nested datatype that can represent slots of differing types. Components: | |
/// | |
/// 1. [`Field`] for each possible child type the Union can hold | |
/// 2. The corresponding `type_id` used to identify which Field | |
/// 3. The type of union (Sparse or Dense) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Updated as suggested.
@@ -516,24 +516,15 @@ impl DataType { | |||
.as_array() | |||
.unwrap() | |||
.iter() | |||
.map(|t| t.as_i64().unwrap()) | |||
.map(|t| t.as_i64().unwrap() as i8) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe we could call t.as_i8()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there is no as_i8()
API for Value
. 😢
@@ -632,39 +632,13 @@ fn array_from_json( | |||
let array = MapArray::from(array_data); | |||
Ok(Arc::new(array)) | |||
} | |||
DataType::Union(fields, _) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 This certainly makes it easier to understand
@@ -435,19 +435,10 @@ mod tests { | |||
"my_union", | |||
DataType::Union( | |||
vec![ | |||
Field::new("f1", DataType::Int32, true).with_metadata(Some( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why was the metadata
removed from this test? Is it simply no longer necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea, it is used to store type id. Now it is stored directly in union type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because we have no place to keep the type ids from Json, I put them in metadata in previously. Now we can keep them in union type formally.
Which issue does this PR close?
Closes #1690.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?