Skip to content

Move filtered SMJ right join out of join_partial phase #13053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 22, 2024
Merged

Move filtered SMJ right join out of join_partial phase #13053

merged 3 commits into from
Oct 22, 2024
+122 −195

Conversation

@comphead
Copy link
Contributor

@comphead comphead commented Oct 22, 2024

Which issue does this PR close?

Related #12359 .
Follow up on #12764

Rationale for this change

Same as #12764 but for Right Outer join

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 22, 2024
comphead
comphead commented Oct 22, 2024
View reviewed changes
datafusion/physical-plan/src/joins/sort_merge_join.rs
join_type: JoinType,
indices: &UInt64Array,
ids: &[usize],
row_indices: &UInt64Array,
Copy link
Contributor Author

@comphead comphead Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed variables for readability

comphead
comphead commented Oct 22, 2024
View reviewed changes
datafusion/sqllogictest/test_files/sort_merge_join.slt
Alice 50 Alice 2

# Uncomment when filtered RIGHT moved
# right join with join filter
Copy link
Contributor Author

@comphead comphead Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reenabling tests

@comphead comphead requested review from alamb and korowa October 22, 2024 15:42
alamb
alamb reviewed Oct 22, 2024
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead -- the only question I have is why some of the tests are removed

datafusion/core/tests/fuzz_cases/join_fuzz.rs
Some(Box::new(col_lt_col_filter)),
)
.run_test(&[JoinTestType::NljHj], false)
.run_test(&[JoinTestType::HjSmj, JoinTestType::NljHj], false)
Copy link
Contributor

@alamb alamb Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

datafusion/physical-plan/src/joins/sort_merge_join.rs

#[tokio::test]
async fn test_left_semi_join_filtered_mask() -> Result<()> {
let schema = Arc::new(Schema::new(vec![
Copy link
Contributor

@alamb alamb Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these tests removed?

Copy link
Contributor Author

@comphead comphead Oct 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are not removed, the common test data preparation was factored out to the separate helper method

alamb
alamb approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @comphead

datafusion/physical-plan/src/joins/sort_merge_join.rs
if matches!(self.join_type, JoinType::Left | JoinType::LeftSemi) {
if matches!(
self.join_type,
JoinType::Left | JoinType::LeftSemi | JoinType::Right
Copy link
Contributor

@alamb alamb Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also have to handle https://docs.rs/datafusion/latest/datafusion/common/enum.JoinType.html#variant.RightSemi 🤔

Copy link
Contributor Author

@comphead comphead Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the end of the day all types of filtered joins will be moved.
I'm planning to cover this week RightSemi, LeftAnti, RightAnti and Full, lets see how it goes

@comphead comphead merged commit afecd7b into apache:main Oct 22, 2024
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb approved these changes

@korowa korowa Awaiting requested review from korowa

Assignees

No one assigned

Labels

core Core DataFusion crate physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@comphead @alamb