Skip to content

perf: specialize single-column hash-join key equality check - #24067

Draft
shehab-ali wants to merge 5 commits into
apache:mainfrom
shehab-ali:shehab/shehab/hashjoin-filter-keys-single-column
Draft

perf: specialize single-column hash-join key equality check#24067
shehab-ali wants to merge 5 commits into
apache:mainfrom
shehab-ali:shehab/shehab/hashjoin-filter-keys-single-column

Conversation

@shehab-ali

@shehab-ali shehab-ali commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Hash-join key-equality verification (equal_rows_arr) checks every candidate row pair produced by a hash-bucket match. The current implementation calls JoinKeyComparator::is_equal per pair, which for each pair:

  1. invokes a boxed DynComparator closure (an un-inlinable indirect call)
  2. computes a full three-way Ordering, only to discard everything except equal / not-equal.

For single-column, high-fanout joins with near-100% match rates (long string keys are the worst case), that per-pair overhead dominates the join and measurably regresses throughput relative to the previous vectorized approach.

Benchmarking against main on the hj SQL benchmark suite (TPC-H SF10, interleaved runs to cancel drift) showed this cost concentrated in single-column-key queries:

  • Q23 (a high-fanout ~28-char string key) ran ~5% slower, with a cluster of other single/low-column joins 6–22% slower.

What changes are included in this PR?

  • Add equal_rows_single_col: a specialized fast path for single-column join keys. It downcasts the key array once and runs a monomorphized == loop over the candidate pairs, eliminating the per-pair boxed-closure dispatch and the Ordering computation.
  • Covered types: integers, Decimal128, dates, timestamps, boolean, and all Utf8/Binary variants (incl. Utf8View/BinaryView).
  • Multi-column keys and unspecialized types fall through unchanged to the existing JoinKeyComparator path. Floats (Float16/32/64) are intentionally excluded so their -0.0/NaN equality semantics remain on the exact same code path as before.
  • Null handling is preserved: NullEqualsNothing -> both-null is unequal; NullEqualsNull -> both-null is equal.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 3, 2026
@codecov-commenter

codecov-commenter commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.61111% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.90%. Comparing base (1955d5a) to head (b2eef82).
⚠️ Report is 63 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/utils.rs 98.61% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24067      +/-   ##
==========================================
+ Coverage   80.75%   80.90%   +0.14%     
==========================================
  Files        1096     1102       +6     
  Lines      373582   376357    +2775     
  Branches   373582   376357    +2775     
==========================================
+ Hits       301686   304482    +2796     
+ Misses      53893    53769     -124     
- Partials    18003    18106     +103     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants