Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-37828][SQL] Push down filters through RebalancePartitions
### What changes were proposed in this pull request? Push down filters through RebalancePartitions. How to reproduce this issue: ```scala spark.sql("SELECT * FROM (SELECT /*+ REBALANCE */ * FROM range(10)) t1 WHERE id = 3").explain(true) ``` Output: ``` == Optimized Logical Plan == Filter (id#0L = 3) +- RebalancePartitions +- Range (0, 10, step=1, splits=None) ``` After this pr: ``` RebalancePartitions +- Filter (id#0L = 3) +- Range (0, 10, step=1, splits=None) ``` ### Why are the changes needed? Improve query performance. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. Closes apache#35118 from wangyum/SPARK-37828. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang <yumwang@ebay.com>
- Loading branch information