Skip to content

perf: Emit LeftSemi hash join rows while probing - #24794

Draft
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:perf/left-semi-incremental
Draft

perf: Emit LeftSemi hash join rows while probing#24794
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:perf/left-semi-incremental

Conversation

@Dandandan

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Rationale for this change

A LeftSemi hash join emits nothing while probing. adjust_indices_by_join_type
returns empty index arrays for it, and every matched build row comes out at the
end from the visited bitmap, the same way LeftAnti and LeftMark work.

HashJoinExec::compute_properties nevertheless reports
EmissionType::Incremental for LeftSemi, grouped with Inner and
RightSemi under the comment "If we only need to generate matched rows from
the probe side". That is not what LeftSemi does: its rows come from the build
side. LeftAnti and LeftMark are correctly classified Both.

So the join is blocking, and a LIMIT above it buys nothing. On TPC-DS SF1:

select ws_order_number from web_sales ws
where exists (select 1 from web_returns wr
              where wr.wr_order_number = ws.ws_order_number)
limit 10
plans as LeftSemi, with LIMIT 10 9.2 ms
the same query with no LIMIT at all 8.6 ms
mirrored so it plans as RightSemi, with LIMIT 10 2.8 ms

What changes are included in this PR?

A semi join only needs to know whether a build row has matched yet, and the
visited bitmap already carries that. This emits a build row when its bit flips
from unset to set, which produces the same rows in the same order but while
probing rather than after it. The final stage then has nothing left to do.

The comment at that site already described the fix: "When visit the right
batch, we can output the matched left row and don't need to wait the end of
loop".

The query above now takes 2.7 ms, matching the RightSemi form, and
EmissionType::Incremental becomes true rather than aspirational.

Only the hash join changes. NestedLoopJoinExec still emits its LeftSemi
rows at the end, so need_produce_result_in_final is untouched.

Are these changes tested?

Yes, by the existing coverage: the full sqllogictest suite (504 files) and the
1117 join unit tests pass. Semi joins are heavily covered by joins.slt,
subquery.slt and the hash join's own tests, including across partition modes
and batch sizes.

One expectation changed, in push_down_filter_parquet.slt. It is an EXPLAIN ANALYZE whose join reports input_batches=1, input_rows=2 where it used to
report 2 and 4; output_rows is 2 either way. The join no longer holds its
output back, so the consumer finishes and the probe scan stops earlier.

Are there any user-facing changes?

LeftSemi hash joins are no longer blocking, so a LIMIT or any other
early-terminating consumer above one can now stop it early. Results are
unchanged.

A `LeftSemi` hash join emitted nothing while probing: every matched build row
came out at the end from the visited bitmap, like `LeftAnti` and `LeftMark`.
`HashJoinExec` nevertheless reported `EmissionType::Incremental` for it,
grouped with Inner and RightSemi under "If we only need to generate matched
rows from the probe side" -- which is not what LeftSemi does, since its rows
come from the build side.

So the join was blocking, and a LIMIT above it bought nothing:

  select ws_order_number from web_sales ws
  where exists (select 1 from web_returns wr
                where wr.wr_order_number = ws.ws_order_number)
  limit 10

took 9.2 ms on TPC-DS SF1, against 8.6 ms for the same query with no LIMIT at
all. Mirrored so it plans as `RightSemi`, which is probe-driven, it took 2.8 ms.

A semi join only needs to know whether a build row has matched yet, and the
bitmap already carries that. Emitting a build row when its bit flips from unset
to set produces the same rows in the same order, while probing. The final stage
then has nothing left to do. The comment at the site said as much already:
"When visit the right batch, we can output the matched left row and don't need
to wait the end of loop".

The query above now takes 2.7 ms, matching the RightSemi form.

Only the hash join changes. `NestedLoopJoinExec` keeps emitting LeftSemi rows
at the end, so `need_produce_result_in_final` is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgqwvctZdvR1ZCz2hbkEJC
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Aug 30, 2026
@codecov-commenter

codecov-commenter commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.52%. Comparing base (61bf6b9) to head (9b9d1d7).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs 95.83% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #24794   +/-   ##
=======================================
  Coverage   81.52%   81.52%           
=======================================
  Files        1123     1123           
  Lines      405970   406054   +84     
  Branches   405970   406054   +84     
=======================================
+ Hits       330978   331051   +73     
- Misses      55627    55639   +12     
+ Partials    19365    19364    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants