[improvement](fe) Convert inner join to semi join when right side is an existence filter - #66535
Open
englefly wants to merge 1 commit into
Open
[improvement](fe) Convert inner join to semi join when right side is an existence filter#66535englefly wants to merge 1 commit into
englefly wants to merge 1 commit into
Conversation
…an existence filter ### What problem does this PR solve? Issue Number: N/A Problem Summary: For queries like `select distinct a1.* from a1, a5 where a1.lot_id = a5.lot_id and a1.ope_no = a5.ope_no and ...` where the right side table of an inner join only appears in equal join conditions (never in the output or any other expression), the inner join acts purely as an existence filter: it only decides which left rows are kept. Such an inner join can be safely rewritten to a left semi join, which keeps the output row count at the left side cardinality instead of multiplying it by the average number of right side matches, and lets the right side scan/broadcast only the join key columns. The conversion is guarded by three conditions, all checked by the new `ConvertInnerJoinToSemiJoin` RBO rule: 1. the right side columns of the join are not referenced above the join (the aggregate/project above only consumes left side columns); 2. all join conditions are equal conjuncts (hashJoinConjuncts non-empty, otherJoinConjuncts empty; `<=>` NullSafeEqual is also covered since it is an EqualPredicate extracted into hashJoinConjuncts by FindHashConditionForJoin); 3. there is a deduplication guarantee above the join (a DISTINCT-like aggregate whose group-by keys cover exactly its output columns), so the row multiplication of an inner join does not change the final result. The rule is registered in the RBO rewrite phase "eliminate join according unique or foreign key" (after infer predicate / push down distinct, before push down limit). ### Release note None ### Check List (For Author) - Test: Unit Test (ConvertInnerJoinToSemiJoinTest, 6 cases: convert with DISTINCT, convert with `<=>`, not convert when right columns leak / no distinct / non-equi condition / aggregate function; sibling join-rule tests pass, no regression) - Behavior changed: No - Does this need documentation: No
englefly
requested review from
924060929,
morrySnow and
starocean999
as code owners
August 6, 2026 07:24
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29182 ms |
Contributor
TPC-DS: Total hot run time: 166473 ms |
Contributor
ClickBench: Total hot run time: 23.71 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29124 ms |
Contributor
TPC-DS: Total hot run time: 159320 ms |
Contributor
ClickBench: Total hot run time: 23.9 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Problem Summary: For queries like
select distinct a1.* from a1, a5 where a1.lot_id = a5.lot_id and a1.ope_no = a5.ope_no and ...where the right side table of an inner join only appears in equal join conditions
(never in the output or any other expression), the inner join acts purely as an
existence filter: it only decides which left rows are kept. Such an inner join can
be safely rewritten to a left semi join, which keeps the output row count at the
left side cardinality instead of multiplying it by the average number of right side
matches, and lets the right side scan/broadcast only the join key columns.
The conversion is guarded by three conditions, all checked by the new
ConvertInnerJoinToSemiJoinRBO rule:(the aggregate/project above only consumes left side columns);
otherJoinConjuncts empty;
<=>NullSafeEqual is also covered since it is anEqualPredicate extracted into hashJoinConjuncts by FindHashConditionForJoin);
whose group-by keys cover exactly its output columns), so the row multiplication
of an inner join does not change the final result.
The rule is registered in the RBO rewrite phase "eliminate join according unique or
foreign key" (after infer predicate / push down distinct, before push down limit).
Release note
None
Check List (For Author)
convert with
<=>, not convert when right columns leak / no distinct / non-equicondition / aggregate function; sibling join-rule tests pass, no regression)