fix: do not prune ORDER BY keys using nullable UNIQUE constraints - #23918
fix: do not prune ORDER BY keys using nullable UNIQUE constraints#23918buraksenn wants to merge 1 commit into
Conversation
| #[derive(Debug, Clone, PartialEq, Eq)] | ||
| pub struct FunctionalDependencies { | ||
| deps: Vec<FunctionalDependence>, | ||
| null_equalities: Vec<NullEquality>, |
There was a problem hiding this comment.
this can be added in FunctionalDependence but since that is pub as well I did not want to change this. I can add this field into that struct if that makes more sense
There was a problem hiding this comment.
Personally it makes more sense to me to add it to FunctionalDependence. Although I imagine these arrays will be the same size and the deps[0] will refer to null_equalities[0]. I think it reads awkwardly
github-actions
Bot
added
core
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23918 +/- ##
=========================================
Coverage 80.65% 80.66%
=========================================
Files 1092 1095 +3
Lines 371140 372344 +1204
Branches 371140 372344 +1204
=========================================
+ Hits 299353 300361 +1008
- Misses 53933 54063 +130
- Partials 17854 17920 +66 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
There are a bunch of related bugs / issues here:
As well as the issue this PR targets, #23818. Based on #23636 (review) , I would guess there might also be an issue with window queries. Probably makes sense to look at this whole area holistically. I see that @alamb and @LDM-A have also been working on related tickets, so we should probably make sure we coordinate. |
Thanks for the heads up. I thought with the tests added in #23821 I can implement a fix but did not see the discussion in #23820. I'll follow it and can also close this PR if it will implemented in a combined fashion |
I have not had too much chance to look into the #23820 ticket after my initial weekend. However @alamb mentioned a refactor of the functional dependency file to take care off all the free functions in it. Maybe since there is multiple PRs coming making changes we can do a refactor then add in the changes. Rather than merging some now then refactoring those with the refactor and finishing some off |
| .strip_backtrace(); | ||
|
|
||
| insta::assert_snapshot!(e, @r#"Error during planning: Extension planner for NoOp created an ExecutionPlan with mismatched schema. LogicalPlan schema: DFSchema { inner: Schema { fields: [Field { name: "a", data_type: Int32 }], metadata: {} }, field_qualifiers: [None], functional_dependencies: FunctionalDependencies { deps: [] } }, ExecutionPlan schema: Schema { fields: [Field { name: "b", data_type: Int32 }], metadata: {} }"#); | ||
| insta::assert_snapshot!(e, @r#"Error during planning: Extension planner for NoOp created an ExecutionPlan with mismatched schema. LogicalPlan schema: DFSchema { inner: Schema { fields: [Field { name: "a", data_type: Int32 }], metadata: {} }, field_qualifiers: [None], functional_dependencies: FunctionalDependencies { deps: [], null_equalities: [] } }, ExecutionPlan schema: Schema { fields: [Field { name: "b", data_type: Int32 }], metadata: {} }"#); |
There was a problem hiding this comment.
For this as well if we make NullEquality part of FunctionalDependence instead of FunctionalDependencies it would create an easier to read error in my opinion
4 participants
Which issue does this PR close?
Rationale for this change
Please check the issue for details, but the main idea is that a nullable
UNIQUEcolumn permits multiple NULL rows, so it does not determine later sort keys across them and they must not be pruned.What changes are included in this PR?
FunctionalDependenciestracks aNullEqualityper dependency. Sort-key pruning now requires the determinant to treat NULLs as equal (GROUP BY key), be non-nullable, or haveNOT NULLsources.Are these changes tested?
Yes, regression tests added in
functional_dependencies.slt.Are there any user-facing changes?
No API changes; incorrect results are fixed.