feat(schema): allow null fields to be merged with other datatypes #4902
Conversation
| @@ -494,7 +497,9 @@ impl Field { | |||
| | DataType::LargeUtf8 | |||
| | DataType::Decimal128(_, _) | |||
| | DataType::Decimal256(_, _) => { | |||
| if self.data_type != from.data_type { | |||
| if from.data_type == DataType::Null { | |||
There was a problem hiding this comment.
I'm not sure about this change as a NullArray is a physically distinct type and is therefore not inherently compatible with any other types. It can be coerced to the same type, via the cast kernels, but it is not inherently compatible which is the notion encoded in this method.
Perhaps https://docs.rs/arrow-cast/latest/arrow_cast/cast/fn.can_cast_types.html can meet your requirements?
|
I see, in C++ API this behavior is guarded by options flag: https://arrow.apache.org/docs/cpp/api/datatype.html#_CPPv4N5arrow5Field12MergeOptionsE |
There was a problem hiding this comment.
Revisiting this again, I was conflating this API with Schema::contains and friends. This change makes sense to me, as this method isn't actually returning a notion of interchangeability, in fact it will even re-order struct fields, which is something the cast kernel can't currently handle (#4908) .
Which issue does this PR close?
Closes #4901
Rationale for this change
All fields can be marked
nullable, so they could be easily made compatible with a field withDataType::NullWhat changes are included in this PR?
Handle
DataType::Nullin schema field mergeAre there any user-facing changes?
Merging null fields no longer returns error when being merged