[SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

[shahidki31]

What changes were proposed in this pull request?

Explain cost command in spark currently doesn't show statistics for all the nodes. It misses some nodes in almost all the TPCDS queries.
In this PR, we are collecting all the plan nodes including the subqueries and computing the statistics for each node, if it doesn't exists in stats cache,

Why are the changes needed?

Before Fix
For eg: Query1, Project node doesn't have statistics

Query15, Aggregate node doesn't have statistics

After Fix:
Query1:

Query 15:

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual testing

[shahidki31]

cc @HyukjinKwon @maropu @cloud-fan @srowen Kindly review

[srowen]

I don't know enough to review this, sorry

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43607/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43607/

[SparkQA]

Test build #139086 has finished for PR 32704 at commit f1eced2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Does this fix nested subqueries?

[shahidki31]

@cloud-fan Yes, collectWithSubqueries will include nested subqueries as well.

[SparkQA]

Test build #139103 has finished for PR 32704 at commit b78a9c9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

Retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43623/

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43627/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43623/

[SparkQA]

Test build #139107 has finished for PR 32704 at commit b78a9c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!

[shahidki31]

Thanks a lot @cloud-fan