fix: inferred predicates for null-aware anti joins#23411
Open
847850277 wants to merge 2 commits into
Open
Conversation
github-actions
Bot
added
optimizer
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes incorrect SQL NOT IN semantics when the subquery can produce NULL by preventing inferred predicate pushdown across null-aware left anti joins (which could otherwise filter out right-side NULLs and change results).
Changes:
- Disable join predicate inference for
JoinType::LeftAntijoins markednull_aware. - Add a sqllogictest regression covering
NOT INwith a right-sideNULLplus an additional left-side filter.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| datafusion/optimizer/src/push_down_filter.rs | Skips inferred predicate generation for null-aware left anti joins to preserve NOT IN null semantics. |
| datafusion/sqllogictest/test_files/not_in_null_pushdown.slt | Adds regression test ensuring right-side NULL is not removed by inferred predicate pushdown. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+592
to
+596
| // Null-aware anti joins implement NOT IN semantics, where NULLs on the | ||
| // right side affect the result. Inferring predicates across the join can | ||
| // incorrectly filter those NULLs. | ||
| if join_type == JoinType::LeftAnti && join.null_aware { | ||
| return Ok(vec![]); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
NOT INmust preserve SQL NULL semantics. If the subquery result containsNULL, then comparisons likea NOT IN (SELECT k FROM u)can evaluate toUNKNOWN, and rows should not be returned byWHERE.DataFusion rewrites
NOT INinto a null-aware anti join. However, predicate pushdown inferred a predicate across that join and could filter out right-sideNULLvalues, changing the query result incorrectly.What changes are included in this PR?
This PR avoids inferring predicates across null-aware left anti joins.
It also adds a sqllogictest regression case covering
NOT INwith a right-sideNULLand an additional left-side filter.Are these changes tested?
Yes.
cargo fmt --all git diff --check cargo test --profile=ci --test sqllogictests -- not_in_null_pushdown.slt cargo clippy --all-targets --all-features -- -D warningsAre there any user-facing changes?