[SPARK-36414][SQL] Disable timeout for BroadcastQueryStageExec in AQE

[yaooqinn]

What changes were proposed in this pull request?

This reverts SPARK-31475, as there are always more concurrent jobs running in AQE mode, especially when running multiple queries at the same time. Currently, the broadcast timeout does not record accurately for the BroadcastQueryStageExec only, but also including the time waiting for being scheduled. If all the resources are currently being occupied for materializing other stages, it timeouts without a chance to run actually.

 

 

The default value is 300s, and it's hard to adjust the timeout for AQE mode. Usually, you need an extremely large number for real-world cases. As you can see in the example, above, the timeout we used for it was 1800s, and obviously, it needed 3x more or something

 

Why are the changes needed?

AQE is default now, we can make it more stable with this PR

Does this PR introduce any user-facing change?

yes, broadcast timeout now is not used for AQE

How was this patch tested?

modified test

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46547/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46547/

[SparkQA]

Test build #142037 has started for PR 33636 at commit b5870e6.

[SparkQA]

Test build #142035 has finished for PR 33636 at commit 88be9e9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46549/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46549/

[HyukjinKwon]

retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46568/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46568/

[cloud-fan]

makes sense to me, cc @maryannxue

[SparkQA]

Test build #142057 has finished for PR 33636 at commit b5870e6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master/3.2!

[dongjoon-hyun]

Could you make a backport to branch-3.1 please, @yaooqinn ?

[yaooqinn]

Could you make a backport to branch-3.1 please, @yaooqinn ?

OK @dongjoon-hyun

[AngersZhuuuu]

Could you make a backport to branch-3.1 please, @yaooqinn ?

OK @dongjoon-hyun

Seems missed?

[dongjoon-hyun]

@AngersZhuuuu . It's too late if this is not backported already.
branch-3.1 is EOL as of today.

[AngersZhuuuu]

@AngersZhuuuu . It's too late if this is not backported already. branch-3.1 is EOL as of today.

Yea