[SPARK-30186][SQL] support Dynamic Partition Pruning in Adaptive Execution

[chrysan]

What changes were proposed in this pull request?

To support queries with dynamic partition pruning subqueries work in adaptive query execution.

Why are the changes needed?

Queries' performance can benefit from AE and DPP at the same time.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Test cases are added.

[wangyum]

ok to test

[SparkQA]

Test build #115039 has finished for PR 26814 at commit f39cc6a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class InsertAdaptiveSparkPlan(

[HyukjinKwon]

@maryannxue FYI

[wangyum]

Also cc @cloud-fan

[maryannxue]

The plan is due to change over the entire process of query execution, so planning the dynamic filters for the whole plan in each re-optimization iteration is definitely the least efficient way. A join can swing from a SMJ to a BHJ or the other way around from time to time, and that affects the applicability of a SubqueryBroadcast DPP filter, and you may end up changing and replanning the filter back and forth.

[chrysan]

The plan is due to change over the entire process of query execution, so planning the dynamic filters for the whole plan in each re-optimization iteration is definitely the least efficient way. A join can swing from a SMJ to a BHJ or the other way around from time to time, and that affects the applicability of a SubqueryBroadcast DPP filter, and you may end up changing and replanning the filter back and forth.

Agree with your concern. But in our real production cases, the re-planning effort in AE is worth if DPP can benefit the pruning a lot. I think we can add a switch to let users turn DPP on in AE.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!