[SPARK-35077][SQL] Migrate to transformWithPruning for leftover optimizer rules

[sigmod]

What changes were proposed in this pull request?

Migrate to transformWithPruning for the following queries:

• SimplifyExtractValueOps
• NormalizeFloatingNumbers
• PushProjectionThroughUnion
• PushDownPredicates
• ExtractPythonUDFFromAggregate
• ExtractPythonUDFFromJoinCondition
• ExtractGroupingPythonUDFFromAggregate
• ExtractPythonUDFs
• CleanupDynamicPruningFilters

Why are the changes needed?

Reduce the number of tree traversals and hence improve the query compilation latency.

How was this patch tested?

Existing tests.
Performance diff:
<style type="text/css"></style>

	Baseline	Experiment	Experiment/Baseline
SimplifyExtractValueOps	99367049	3679579	0.04
NormalizeFloatingNumbers	24717928	20451094	0.83
PushProjectionThroughUnion	14130245	7913551	0.56
PushDownPredicates	276333542	261246842	0.95
ExtractPythonUDFFromAggregate	6459451	2683556	0.42
ExtractPythonUDFFromJoinCondition	5695404	2504573	0.44
ExtractGroupingPythonUDFFromAggregate	5546701	1858755	0.34
ExtractPythonUDFs	58726458	1598518	0.03
CleanupDynamicPruningFilters	26606652	15417936	0.58
OptimizeSubqueries	3072287940	2876462708	0.94

[SparkQA]

Test build #139133 has finished for PR 32721 at commit d95d332.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43653/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43655/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43655/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43663/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43663/

[SparkQA]

Test build #139135 has finished for PR 32721 at commit 864ee6f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #139143 has finished for PR 32721 at commit 64220b7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sigmod]

@dbaliafroozeh @gengliangwang @hvanhovell @maryannxue this PR is ready for review.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43695/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43695/

[SparkQA]

Test build #139175 has finished for PR 32721 at commit 58732f9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

Merging to master