Support join filter for `SortMergeJoin` #9080

viirya · 2024-01-31T08:11:18Z

Which issue does this PR close?

Closes #.

Rationale for this change

DataFusion SortMergeJoin doesn't support join filter for now. Any logical join operator with join filter could only be planned as HashJoinExec which supports join filter.

Spark SortMergeJoin supports join filter. Without join filter support, we cannot translate Spark SortMergeJoin operator to DataFusion.

What changes are included in this PR?

This patch adds join filter support to DataFusion SortMergeJoin.

Are these changes tested?

Are there any user-facing changes?

alamb

Thanks @viirya -- the code is looking pretty good to me. I think this PR may need to handle OUTER joins as well, but maybe it already does. Adding some test coverage could probably tell us one way or the other

alamb · 2024-01-31T21:15:14Z

datafusion/sqllogictest/test_files/sort_merge_join.slt

+# under the License.
+
+##########
+## Sort Merge Join Tests


I think we should also add tests for LEFT/RIGHT OUTER joins where the filter needs to be applied to the non - preserved side (aka applied during the join)

Yes, all supported Join types are supported. Let me add more test coverage.

ozankabak · 2024-01-31T21:25:10Z

PTAL @metesynnada

metesynnada

I believe there should be more unit tests for numeric values as well. Other than that, the PR looks fine.

metesynnada · 2024-02-01T08:36:57Z

datafusion/physical-plan/src/joins/sort_merge_join.rs

            let mut streamed_columns = self
                .streamed_schema
                .fields()
                .iter()
                .map(|f| new_null_array(f.data_type(), buffered_indices.len()))
                .collect::<Vec<_>>();

+            let filter_columns =


Inside get_filter_column, the filter is checked if it is Some. If not, the result will be an empty vector.

Instead of doing that, you can move the filter columns calculation under
let output_batch = if let Some(f) = &self.filter { and make get_filter_column expects joinfilter_filter: &JoinFilter.

It is because buffered_columns is consumed by streamed_columns.extend(buffered_columns);. So either I clone buffered_columns before it, or put get_filter_column before it like I current do.

viirya · 2024-02-01T17:29:44Z

Thanks for review. Found this has some issues on OUTER joins, going to revise this and add more tests.

alamb · 2024-02-02T21:52:25Z

Marking as a draft to make it clear this PR is not waiting on review

viirya · 2024-02-03T02:00:15Z

datafusion/sqllogictest/test_files/join.slt

+33 c 3 NULL NULL NULL
+44 d 4 44 x 3
+NULL NULL NULL 11 z 3
+NULL NULL NULL 55 w 3


This is added to compare with sort_merge_join.slt results for full join.

viirya · 2024-02-03T02:17:00Z

Thank you @alamb. I fixed the issue for outer joins and marked this ready for review again.

alamb · 2024-02-05T11:31:21Z

Thanks @viirya -- I plan to review this later today

alamb

Thank you very much @viirya -- I reviewed the code carefully and it looks like a nice improvement to me and has good test coverage. I had some code organization / comment suggestions, but nothing that I think would prevent merging

cc @korowa, @liukun4515 and @metesynnada in case you have some additional thoughts to share as I think you may be famililar with this code.

alamb · 2024-02-05T15:22:38Z

datafusion/physical-plan/src/joins/sort_merge_join.rs

+                RecordBatch::try_new(self.schema.clone(), columns.clone())?;
+
+            // Apply join filter if any
+            if !filter_columns.is_empty() {


I don't undersand why there is the check for filer columns and if self.filter is Some. I expected the check to simply be if self.filter is some (and the else case is the same for both below)

If the filter has no columns, it seems like the else clause does the same thing in both cases.

Thus, I wonder if we could remove the check for filter_columns entirely 🤔

Because if this joined batch is between streamed batch and null (i.e., outer joins), we don't need to handle join filter (although join filter is Some).

I will add a short comment here.

Ah, I see -- this would be for batches that don't have any matches from equality predicates anyways - which makes sense

alamb · 2024-02-05T15:22:42Z

datafusion/physical-plan/src/joins/sort_merge_join.rs

+                        self.join_type,
+                        JoinType::Left | JoinType::Right | JoinType::Full
+                    ) {
+                        // The reverse of the selection mask, which is for null joined rows


Does 'null joined rows' mean 'rows that passed neither the equijoin predicates NOR the filter? If so I would find a term like 'non_matching_rows` easier to understand. But that is a personal preference

The rows reaches here are all passed the equijoin predicates already (their buffered_batch_idx is Some). "null joined rows" here means the rows not pass the join filter, and we are going to join them (left or right side) with null. Let me add a few words to make it clear.

alamb · 2024-02-05T15:22:45Z

datafusion/physical-plan/src/joins/sort_merge_join.rs

+
+                            buffered_columns.extend(streamed_columns);
+                            buffered_columns
+                        } else {


I missed the fact that this handles left and full (not just left)

Suggested change

} else {

}

// Left join or full outer join

else {

alamb · 2024-02-05T15:23:20Z

datafusion/physical-plan/src/joins/sort_merge_join.rs

@@ -1142,12 +1294,49 @@ impl SMJStream {
        let record_batch = concat_batches(&self.schema, &self.output_record_batches)?;
        self.join_metrics.output_batches.add(1);
        self.join_metrics.output_rows.add(record_batch.num_rows());
-        self.output_size -= record_batch.num_rows();
+        // If join filter exists, `self.output_size` is not accurate as we don't know the exact


Is the idea here that output_size is tracking the number of rows remaining to output? If so, it seems like the filter could only decrease the number of output rows (never increase it)

However, I can see how the SMJ code could overshoot for LEFT/RIGHT/FULL joins, so maybe this fix was needed because now there is more test coverage of SMJ 🤔

The logic of output_size assumes that each row put into the buffer will produce exactly one output row. It is increased when we put rows into buffer and decreased after we actually output batches.

So it is used to track the number of rows in buffers. We compare it with self.batch_size (the target output batch size), and decide to output batches from buffers if it reaches.

For joins with join filter cases, the assumption of output_size is broken. One row put into the buffer may produce more than one output row. For example, one joined row under full join doesn't pass join filter, then it will produce two output rows, i.e., streamed row joined null row and null row joined buffered row.

So the actual output rows record_batch.num_rows() may be larger than self.output_size and self.output_size -= record_batch.num_rows() will cause overflow.

For such case, we can simply clean up output_size.

I got it -- thank you for the explanation. I didn't understand the assumptions / invariants of output_size. Maybe we can clarify this somehow in comments (I left one suggestion, but maybe it is not correct)

Thank you @alamb. The suggestion looks good to me.

datafusion/physical-plan/src/joins/sort_merge_join.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

viirya · 2024-02-07T16:49:21Z

Thanks @alamb @metesynnada for review.

viirya added 2 commits January 30, 2024 22:22

Support join filter for SortMergeJoin

6dbfed6

Move test

159c693

github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jan 31, 2024

alamb mentioned this pull request Jan 31, 2024

DataFusion weekly project plan (Andrew Lamb) - Jan 29, 2024 #9030

Closed

6 tasks

Fix test

d9e40af

viirya force-pushed the sort_merge_join_filter branch from 7ab4393 to d9e40af Compare January 31, 2024 20:32

Fix clippy

00d4160

alamb reviewed Jan 31, 2024

View reviewed changes

Add outer join tests

99940c2

metesynnada reviewed Feb 1, 2024

View reviewed changes

alamb marked this pull request as draft February 2, 2024 21:52

Fix outer join

be35c90

viirya commented Feb 3, 2024

View reviewed changes

viirya marked this pull request as ready for review February 3, 2024 02:16

alamb mentioned this pull request Feb 5, 2024

DataFusion weekly project plan (Andrew Lamb) - Feb 5, 2024 #9121

Closed

6 tasks

alamb approved these changes Feb 5, 2024

View reviewed changes

For review

c7c25ce

alamb approved these changes Feb 6, 2024

View reviewed changes

datafusion/physical-plan/src/joins/sort_merge_join.rs Outdated Show resolved Hide resolved

Update datafusion/physical-plan/src/joins/sort_merge_join.rs

e75629d

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

viirya merged commit 13fdf89 into apache:main Feb 7, 2024
22 checks passed

This was referenced Feb 8, 2024

Use prep_null_mask_filter to handle nulls in selection mask #9163

Merged

Join support apache/datafusion-comet#12

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support join filter for `SortMergeJoin` #9080

Support join filter for `SortMergeJoin` #9080

viirya commented Jan 31, 2024

alamb left a comment

alamb Jan 31, 2024

viirya Jan 31, 2024

ozankabak commented Jan 31, 2024

metesynnada left a comment

metesynnada Feb 1, 2024

viirya Feb 1, 2024

viirya commented Feb 1, 2024

alamb commented Feb 2, 2024

viirya Feb 3, 2024

viirya commented Feb 3, 2024

alamb commented Feb 5, 2024

alamb left a comment

alamb Feb 5, 2024

viirya Feb 5, 2024

viirya Feb 5, 2024

alamb Feb 5, 2024

alamb Feb 5, 2024

viirya Feb 5, 2024

alamb Feb 5, 2024

alamb Feb 5, 2024

viirya Feb 5, 2024

alamb Feb 6, 2024

viirya Feb 6, 2024

viirya commented Feb 7, 2024

Support join filter for SortMergeJoin #9080

Support join filter for SortMergeJoin #9080

Conversation

viirya commented Jan 31, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ozankabak commented Jan 31, 2024

metesynnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Feb 1, 2024

alamb commented Feb 2, 2024

Choose a reason for hiding this comment

viirya commented Feb 3, 2024

alamb commented Feb 5, 2024

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Feb 7, 2024

Support join filter for `SortMergeJoin` #9080

Support join filter for `SortMergeJoin` #9080