More properly handle nullability of types/literals in Substrait #10640
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Blizzara
force-pushed
the
avo/substrait-nullability
branch
from
f475c78
to
6a1c0cb
Compare
This isn't perfect; some things are still assumed to just always be nullable (e.g. Literal list elements). But it's still giving a closer match than just assuming everything is nullable.
Blizzara
force-pushed
the
avo/substrait-nullability
branch
from
6a1c0cb
to
76925d0
Compare
jonahgao
reviewed
May 24, 2024
| @@ -1686,7 +1691,7 @@ fn to_substrait_bounds(window_frame: &WindowFrame) -> Result<(Bound, Bound)> { | |||
| )) | |||
| } | |||
|
|
|||
| fn to_substrait_literal(value: &ScalarValue) -> Result<Literal> { | |||
| fn to_substrait_literal(value: &ScalarValue, nullable: bool) -> Result<Literal> { | |||
There was a problem hiding this comment.
According to ExprSchemable::nullable(), the nullability of a literal depends on its ScalarValue, so I think the nullable argument for this function could be omitted.
fn to_substrait_literal(value: &ScalarValue, nullable: bool) -> Result<Literal> {
if value.is_null() {
return Ok(Literal {
nullable: true,
type_variation_reference: DEFAULT_TYPE_REF,
literal_type: Some(LiteralType::Null(to_substrait_type(
&value.data_type(),
true,
)?)),
});
}At the same time, we can also remove try_to_substrait_null.
There was a problem hiding this comment.
I think you're right, thanks! That does indeed simplify things. Fixed in cbd2264
jonahgao
reviewed
May 24, 2024
Co-authored-by: Jonah Gao <jonahgao@msn.com>
Blizzara
force-pushed
the
avo/substrait-nullability
branch
from
cbd2264
to
d9a38b1
Compare
jayzhan211
pushed a commit
to jayzhan211/arrow-datafusion
that referenced
this pull request
May 26, 2024
…he#10640) * More properly handle nullability of types/literals in Substrait This isn't perfect; some things are still assumed to just always be nullable (e.g. Literal list elements). But it's still giving a closer match than just assuming everything is nullable. * Avoid cloning and creating DataFusionError Co-authored-by: Jonah Gao <jonahgao@msn.com> * simplify Literal/ScalarValue null handling --------- Co-authored-by: Jonah Gao <jonahgao@msn.com>
This was referenced May 27, 2024
Blizzara
added a commit
to Blizzara/datafusion
that referenced
this pull request
Jun 11, 2024
DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of apache#10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ```
alamb
pushed a commit
that referenced
this pull request
Jun 12, 2024
…0874) * Ignore nullability of list elements when consuming Substrait DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of #10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ``` * use `Field::new_list_field` in `array_into_(large_)list_array` just for consistency, to reduce the places where "item" is written out * add a test for non-nullable lists
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…he#10640) * More properly handle nullability of types/literals in Substrait This isn't perfect; some things are still assumed to just always be nullable (e.g. Literal list elements). But it's still giving a closer match than just assuming everything is nullable. * Avoid cloning and creating DataFusionError Co-authored-by: Jonah Gao <jonahgao@msn.com> * simplify Literal/ScalarValue null handling --------- Co-authored-by: Jonah Gao <jonahgao@msn.com>
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…ache#10874) * Ignore nullability of list elements when consuming Substrait DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of apache#10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ``` * use `Field::new_list_field` in `array_into_(large_)list_array` just for consistency, to reduce the places where "item" is written out * add a test for non-nullable lists
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Extracted from #10531
Builds on top of #10622 so I'll rebase this once it's merged.
Rationale for this change
More closely propagate null-ability through Substrait conversions
What changes are included in this PR?
Handles the nullability of struct fields during Substrait conversions. This isn't perfect; some things are still assumed to just always be nullable (e.g. Literal list elements).
Are these changes tested?
Tested through round-trip unit tests
Are there any user-facing changes?
Nulls are handled more accurately