[SPARK-24954][Core] Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

[jiangxb1987]

What changes were proposed in this pull request?

We don't support run a barrier stage with dynamic resource allocation enabled, it shall lead to some confusing behaviors (eg. with dynamic resource allocation enabled, it may happen that we acquire some executors (but not enough to launch all the tasks in a barrier stage) and later release them due to executor idle time expire, and then acquire again).

We perform the check on job submit and fail fast if running a barrier stage with dynamic resource allocation enabled.

How was this patch tested?

Added new test suite BarrierStageOnSubmittedSuite to cover all the fail fast cases that submitted a job containing one or more barrier stages.

[holdensmagicalunicorn]

@jiangxb1987, thanks! I am a bot who has found some folks who might be able to help with the review:@mateiz, @rxin and @kayousterhout

[jiangxb1987]

cc @squito

[SparkQA]

Test build #93791 has finished for PR 21915 at commit 2ffa2b7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

• [SPARK-24942]: Barrier execution mode does not support dynamic resource allocation for now. You can disable dynamic resource allocation by setting Spark conf "spark.dynamicAllocation.enabled" to "false".
• Make the error message a constant to simplify test.

[mengxr]

Does it work if we use LocalSparkContext? nvm, I was thinking MLlibTestSparkContext ...

[squito]

thanks @jiangxb1987, lgtm aside from @mengxr 's comments

[SparkQA]

Test build #93877 has finished for PR 21915 at commit 663b900.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[SparkQA]

Test build #93889 has finished for PR 21915 at commit 663b900.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

retest this please

[SparkQA]

Test build #93910 has finished for PR 21915 at commit 663b900.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[mengxr]

LGTM pending jenkins. Was there a JIRA reported about KafkaContinuousSourceSuite failure? cc: @tedyu ?

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93889/testReport/org.apache.spark.sql.kafka010/KafkaMicroBatchV1SourceSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/

[SparkQA]

Test build #93931 has finished for PR 21915 at commit 663b900.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[SparkQA]

Test build #93971 has finished for PR 21915 at commit 663b900.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[SparkQA]

Test build #94006 has finished for PR 21915 at commit 663b900.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #94089 has finished for PR 21915 at commit f3ea9c6.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #94090 has finished for PR 21915 at commit 0796f76.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

test this please

[SparkQA]

Test build #94115 has finished for PR 21915 at commit 0796f76.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[SparkQA]

Test build #94131 has finished for PR 21915 at commit 0796f76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. Thanks!