[SPARK-33027][SQL] Add DisableUnnecessaryBucketedScan rule to AQE #30200
Conversation
|
cc @cloud-fan if you have time to take a look, thanks. |
|
Kubernetes integration test starting |
| removeRedundantSorts, | ||
| ensureRequirements | ||
| ensureRequirements, | ||
| disableUnnecessaryBucketedScan |
There was a problem hiding this comment.
seems we can just put DisableUnnecessaryBucketedScan. Same to other rules.
| val partitioning = spark.table("t1").queryExecution.executedPlan | ||
| .outputPartitioning | ||
| assert(partitioning match { | ||
| val inMemoryScan = find(spark.table("t1").queryExecution.executedPlan)( |
There was a problem hiding this comment.
how about stripAQEPlan(spark.table("t1").queryExecution.executedPlan).outputPartitioning
There was a problem hiding this comment.
@cloud-fan - sure, it's less verbose, updated.
|
Kubernetes integration test status success |
|
Test build #130445 has finished for PR 30200 at commit
|
|
Test build #130440 has finished for PR 30200 at commit
|
|
retest this please |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #130447 has finished for PR 30200 at commit
|
|
thanks, merging to master! |
|
Thank you @cloud-fan for review! |
3 participants
What changes were proposed in this pull request?
As a followup comment from #29804 (comment) , here we add add the physical plan rule DisableUnnecessaryBucketedScan into AQE AdaptiveSparkPlanExec.queryStagePreparationRules, to make auto bucketed scan work with AQE.
The change is mostly in:
AdaptiveSparkPlanExec.scala: add physical plan ruleDisableUnnecessaryBucketedScanDisableUnnecessaryBucketedScan.scala: propagate logical plan link for the file source scan exec operator, otherwise we lose the logical plan link information when AQE is enabled, and will get exception here. (for example, for querySELECT * FROM bucketed_tablewith AQE is enabled)DisableUnnecessaryBucketedScanSuite.scala: add new test suite for AQE enabled -DisableUnnecessaryBucketedScanWithoutHiveSupportSuiteAE, and changed some of tests to useAdaptiveSparkPlanHelper.find/collect, to make the plan verification work when AQE enabled.Why are the changes needed?
It's reasonable to add the support to allow disabling unnecessary bucketed scan with AQE is enabled, this helps optimize the query when AQE is enabled.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added unit test in
DisableUnnecessaryBucketedScanSuite.