Skip to content

[SPARK-39600][SQL] Enhance pushdown limit through window#36990

Closed
wangyum wants to merge 4 commits intoapache:masterfrom
wangyum:SPARK-39600
Closed

[SPARK-39600][SQL] Enhance pushdown limit through window#36990
wangyum wants to merge 4 commits intoapache:masterfrom
wangyum:SPARK-39600

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Jun 25, 2022

What changes were proposed in this pull request?

This PR adds support more cases for pushing down limit through window.

  1. Window's partition specification is not empty. It will push down top N with global sort by partition specification and order specification.
    -- Origin query
    SELECT *, ROW_NUMBER() OVER(PARTITION BY a ORDER BY b) AS rn FROM Tab1 LIMIT 5;
    -- Equivalent push down limit query
    SELECT *,
           row_number() OVER(partition BY a ORDER BY b) AS rn
    FROM   (SELECT * FROM tab1 ORDER BY a, b LIMIT 5) t
  2. It has less than or less than or equal filter condition on that pushable window and limit number less than or equal with that filter condition.
    -- Origin query
    SELECT *
    FROM   (SELECT *, row_number() OVER(PARTITION BY a ORDER BY b) AS rn FROM tab1) t
    WHERE  rn <= 5
    LIMIT 5
    -- Equivalent push down limit query
    SELECT *
    FROM   (SELECT *,
                   row_number() OVER(partition BY a ORDER BY b) AS rn
            FROM   (SELECT * FROM tab1 ORDER BY a, b LIMIT 5) t)
    WHERE rn <= 5

Why are the changes needed?

Improve query performance.

Impala also support it: https://issues.apache.org/jira/browse/IMPALA-9983

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@github-actions github-actions bot added the SQL label Jun 25, 2022
@wangyum wangyum force-pushed the SPARK-39600 branch 2 times, most recently from 763ad64 to cfa653e Compare June 28, 2022 01:37
@wangyum
Copy link
Member Author

wangyum commented Jun 28, 2022

@cloud-fan @sigmod

@HyukjinKwon
Copy link
Member

cc @hvanhovell FYI

@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Oct 15, 2022
@github-actions github-actions bot closed this Oct 16, 2022
@xy-xin
Copy link

xy-xin commented Oct 8, 2023

Will this be continue? @wangyum

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants