Skip to content

Commit

Permalink
[SPARK-48265][SQL] Infer window group limit batch should do constant …
Browse files Browse the repository at this point in the history
…folding

### What changes were proposed in this pull request?
Plan after PropagateEmptyRelation may generate double local limit
```
 GlobalLimit 21
 +- LocalLimit 21
!   +- Union false, false
!      :- LocalLimit 21
!      :  +- Project [item_id#647L]
!      :     +- Filter (xxxx)
!      :        +- Relation db.table[,... 91 more fields] parquet
!      +- LocalLimit 21
!         +- Project [item_id#738L]
!            +- LocalRelation <empty>, [, ... 91 more fields]
```
to
```
 GlobalLimit 21
    +- LocalLimit 21
       - LocalLimit 21
          +- Project [item_id#647L]
            +- Filter (xxxx)
               +- Relation db.table[,... 91 more fields] parquet
```
after `Infer window group limit batch` batch's `EliminateLimits`
will be
```
 GlobalLimit 21
    +- LocalLimit least(21, 21)
          +- Project [item_id#647L]
            +- Filter (xxxx)
               +- Relation db.table[,... 91 more fields] parquet
```
It can't work, here miss a `ConstantFolding`

### Why are the changes needed?
Fix bug

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #46568 from AngersZhuuuu/SPARK-48265.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
AngersZhuuuu authored and cloud-fan committed May 14, 2024
1 parent 0ea8088 commit 7974811
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ class SparkOptimizer(
InferWindowGroupLimit,
LimitPushDown,
LimitPushDownThroughWindow,
EliminateLimits) :+
EliminateLimits,
ConstantFolding) :+
Batch("User Provided Optimizers", fixedPoint, experimentalMethods.extraOptimizations: _*) :+
Batch("Replace CTE with Repartition", Once, ReplaceCTERefWithRepartition)

Expand Down

0 comments on commit 7974811

Please sign in to comment.