branch-4.1: [feature](fe) Push down limit into CTE producer (#63675)#64003
Open
CalvinKirs wants to merge 1 commit into
Open
branch-4.1: [feature](fe) Push down limit into CTE producer (#63675)#64003CalvinKirs wants to merge 1 commit into
CalvinKirs wants to merge 1 commit into
Conversation
This PR adds CTE producer-side limit pushdown in Nereids.
When all CTE consumers only need a bounded number of rows, the optimizer
collects the required row count from each consumer, takes
the maximum value, and pushes that limit into the CTE producer. The
original consumer-side limit is still kept.
The rule only handles safe shapes:
```text
LogicalLimit
LogicalCTEConsumer
```
```text
LogicalLimit
LogicalProject
LogicalCTEConsumer
```
The project must be row-preserving.
## Scenarios
### 1. Direct Limit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10;
```
The consumer only needs 10 rows, so the CTE producer can produce at most
10 rows.
### 2. Project + Limit
```sql
WITH cte AS (
SELECT order_id, total_price, user_id FROM orders
)
SELECT order_id, total_price
FROM cte
LIMIT 10;
```
A normal project only prunes columns and does not change row count, so
the producer can still be limited to 10 rows.
### 3. Multiple Consumers + Limit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte LIMIT 10
UNION ALL
SELECT * FROM cte LIMIT 20;
```
For multiple CTE consumers, the producer limit is:
```text
producerLimit = max(consumerLimit1, consumerLimit2, ...)
```
In this case, the pushed producer limit is 20.
If any consumer needs full CTE data, pushdown is skipped:
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte LIMIT 10
UNION ALL
SELECT * FROM cte;
```
### 4. Limit + Offset
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10 OFFSET 100;
```
The consumer needs to skip 100 rows and then return 10 rows, so the
producer must provide at least 110 rows.
The producer side only truncates rows and does not apply offset:
```text
producerLimit = limit + offset
producerOffset = 0
```
### 5. SplitLimit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10 OFFSET 100;
```
Doris may split this into local/global limits. The local limit closest
to the CTE consumer already represents `limit + offset`.
The collector uses the local limit value directly and does not add
offset again.
### 6. Filter + Limit Is Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
WHERE order_id > 10
LIMIT 10;
```
Filter can reduce rows before limit, so the producer may need more than
10 input rows. This rule does not push limit through filter.
### 7. TopN Is Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
ORDER BY order_id
LIMIT 10;
```
`ORDER BY ... LIMIT` is TopN. It needs the first N rows after ordering,
so it cannot be treated as a normal limit.
### 8. Join / Aggregate / Window / Sort Are Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM cte JOIN users ON cte.user_id = users.user_id
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT user_id, COUNT(*)
FROM cte
GROUP BY user_id
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM (
SELECT order_id, ROW_NUMBER() OVER (ORDER BY order_id) AS rn
FROM cte
) t
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM (
SELECT * FROM cte ORDER BY order_id
) t
LIMIT 10;
```
These operators can change row cardinality or ordering semantics. Unless
other rules have already rewritten the shape into `Limit ->
CTEConsumer` or `Limit -> Project -> CTEConsumer`, this collector skips
them.
(cherry picked from commit d898a1d)
Member
Author
|
run buildall |
1 similar comment
Member
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master PR: #63675