Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-39338][SQL] Remove dynamic pruning subquery if pruningKey's references is empty #36724

Closed
wants to merge 2 commits into from
Closed

[SPARK-39338][SQL] Remove dynamic pruning subquery if pruningKey's references is empty #36724

wants to merge 2 commits into from

Conversation

wangyum
Copy link
Member

@wangyum wangyum commented May 30, 2022 •
edited

What changes were proposed in this pull request?

Remove dynamic pruning subquery if pruningKey's references is empty. For example:

SELECT f.store_id,
       f.date_id,
       s.state_province
FROM (SELECT   store_id,
               date_id,
               product_id
      FROM   fact_stats
      WHERE  date_id <= 1000
      UNION ALL
      SELECT 4 AS store_id,
               date_id,
               product_id
      FROM   fact_sk
      WHERE  date_id >= 1300) f
JOIN dim_store s
ON f.store_id = s.store_id
WHERE s.country IN ('US', 'NL')
Before this PR After this PR
image image

Why are the changes needed?

Remove useless dynamic pruning subquery because it can't reduce partition.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@github-actions github-actions bot added the SQL label May 30, 2022
@wangyum
Copy link
Member Author

wangyum commented May 30, 2022

cc @cloud-fan

cloud-fan
cloud-fan approved these changes May 31, 2022
View reviewed changes
Copy link
Contributor

@cloud-fan cloud-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, cc @sigmod

weixiuli
weixiuli approved these changes May 31, 2022
View reviewed changes
Copy link
Contributor

@weixiuli weixiuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM.

@wangyum wangyum closed this in cb0e419 Jun 1, 2022
@wangyum
Copy link
Member Author

wangyum commented Jun 1, 2022

Merged to master.

@wangyum wangyum deleted the SPARK-39338 branch June 1, 2022 03:38
@wangyum wangyum mentioned this pull request Aug 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sigmod sigmod sigmod approved these changes

@cloud-fan cloud-fan cloud-fan approved these changes

@weixiuli weixiuli weixiuli approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
4 participants
@wangyum @sigmod @cloud-fan @weixiuli