[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming #23845

liupc · 2019-02-20T11:29:49Z

What changes were proposed in this pull request?

Currently, when enabled streaming dynamic allocation for streaming applications, the maxNumExecutorFailures in ApplicationMaster is still computed with spark.dynamicAllocation.maxExecutors.

Actually, we should consider spark.streaming.dynamicAllocation.maxExecutors instead.

Related codes:

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

Line 101 in f87153a

private val maxNumExecutorFailures = {

How was this patch tested?

NA

Please review http://spark.apache.org/contributing.html before opening a pull request.

streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala

core/src/main/scala/org/apache/spark/internal/config/package.scala

streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala

core/src/main/scala/org/apache/spark/internal/config/package.scala

srowen · 2019-02-20T14:50:03Z

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

@@ -100,7 +100,9 @@ private[spark] class ApplicationMaster(

  private val maxNumExecutorFailures = {
    val effectiveNumExecutors =
-      if (Utils.isDynamicAllocationEnabled(sparkConf)) {
+      if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
+        sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)


For other reviewers -- this is the fix itself.

core/src/main/scala/org/apache/spark/internal/config/package.scala

SparkQA · 2019-02-22T15:02:05Z

Test build #4565 has finished for PR 23845 at commit d5a5107.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

srowen · 2019-02-25T10:46:44Z

@liupc could you have a look at the build failure?

SparkQA · 2019-02-26T15:54:28Z

Test build #4575 has finished for PR 23845 at commit 155f284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

liupc · 2019-03-01T02:19:13Z

retest this please!

liupc · 2019-03-01T03:26:35Z

@srowen UT seems failed with some unrelated tests, can you help to retest this please?

SparkQA · 2019-03-01T17:35:46Z

Test build #4583 has finished for PR 23845 at commit 155f284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

liupc · 2019-03-05T06:59:56Z

retest this please

liupc · 2019-03-06T06:46:25Z

retest this please

liupc · 2019-03-06T06:47:47Z

cc @HyukjinKwon @srowen

SparkQA · 2019-03-06T15:19:05Z

Test build #4598 has started for PR 23845 at commit f825118.

SparkQA · 2019-03-09T18:08:20Z

Test build #4603 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-03-11T14:15:20Z

Test build #4608 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

srowen · 2019-03-11T14:24:22Z

Hm, that's weird. The failure is not directly related, and I thought we fixed this in #23887 Not sure what's going on. dev/lint-java and dev/sbt-checkstyle pass locally for master too, and this isn't failing other PR builds. Maybe some odd cached artifacts on a Jenkins build server? no idea.

SparkQA · 2019-03-11T14:37:20Z

Test build #4609 has started for PR 23845 at commit f825118.

SparkQA · 2019-03-12T18:53:38Z

Test build #4612 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

vanzin

I'm really not familiar with streaming dynamic allocation, but I assume you can't enable both that and regular dynamic allocation together?

If for some reason that is allowed, then the fix should take the max of both values.

To fix the tests, probably needs a merge with master. Maybe github is confused.

vanzin · 2019-03-13T00:17:05Z

core/src/main/scala/org/apache/spark/internal/config/package.scala

@@ -332,6 +332,51 @@ package object config {
    ConfigBuilder("spark.dynamicAllocation.sustainedSchedulerBacklogTimeout")
      .fallbackConf(DYN_ALLOCATION_SCHEDULER_BACKLOG_TIMEOUT)

+  private[spark] val STREAMING_DYN_ALLOCATION_ENABLED =


Could you add these to a new Streaming.scala instead? We shouldn't be adding more stuff to package.scala.

@vanzin

If for some reason that is allowed, then the fix should take the max of both values.

This is not allowed to enable both of them. This check is done in the scheduler/ExecutorAllocationManager

spark/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala

Line 211 in 7beb464

throw new IllegalArgumentException(

I agree to put these configs to Streaming.scala, it's more clear. I will update.

vanzin · 2019-03-13T22:13:21Z

ok to test

SparkQA · 2019-03-13T22:26:39Z

Test build #103459 has finished for PR 23845 at commit 2f46380.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-15T18:12:25Z

Test build #4625 has finished for PR 23845 at commit 2f46380.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-03-15T18:30:27Z

@liupc can you take a look at ...

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/src/main/scala/org/apache/spark/util/Utils.scala:2495: not found: value STREAMING_DYN_ALLOCATION_ENABLED
[error]     val streamingDynamicAllocationEnabled = conf.get(STREAMING_DYN_ALLOCATION_ENABLED)
[error]

SparkQA · 2019-03-16T22:43:38Z

Test build #103570 has finished for PR 23845 at commit a46d4d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-03-17T00:45:12Z

merged to master

liupc changed the title ~~[SPARK-26941]Fix incorrect computation of maxNumExecutorFailures for streaming~~ [SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming Feb 20, 2019

srowen requested changes Feb 20, 2019

View reviewed changes

srowen approved these changes Feb 22, 2019

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/package.scala Outdated Show resolved Hide resolved

vanzin reviewed Mar 13, 2019

View reviewed changes

Liupengcheng added 5 commits March 13, 2019 09:30

Fix incorrect computation of maxNumExecutorFailures for streaming

f513285

Replace hardcoded configs with ConfigEntry

1b1e4d0

Refine as commented

ddc5312

fix streaming is not an enclosing class

bd21ba3

Fix ExecutorAllocationManasterSuite

882bd35

liupc force-pushed the Fix-incorrect-maxNumExecutorFailures-for-streaming branch from f825118 to 882bd35 Compare March 13, 2019 01:31

Moving streaming configs to Streaming.scala

2f46380

Add missing Streaming.scala

a46d4d3

srowen closed this in cad475d Mar 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming #23845

[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming #23845

liupc commented Feb 20, 2019

srowen Feb 20, 2019

SparkQA commented Feb 22, 2019

srowen commented Feb 25, 2019

SparkQA commented Feb 26, 2019

liupc commented Mar 1, 2019

liupc commented Mar 1, 2019

SparkQA commented Mar 1, 2019

liupc commented Mar 5, 2019

liupc commented Mar 6, 2019

liupc commented Mar 6, 2019

SparkQA commented Mar 6, 2019

SparkQA commented Mar 9, 2019

SparkQA commented Mar 11, 2019

srowen commented Mar 11, 2019 •

edited

SparkQA commented Mar 11, 2019

SparkQA commented Mar 12, 2019

vanzin left a comment

vanzin Mar 13, 2019

liupc Mar 13, 2019

vanzin commented Mar 13, 2019

SparkQA commented Mar 13, 2019

SparkQA commented Mar 15, 2019

srowen commented Mar 15, 2019

SparkQA commented Mar 16, 2019

srowen commented Mar 17, 2019

[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming #23845

[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming #23845

Conversation

liupc commented Feb 20, 2019

What changes were proposed in this pull request?

How was this patch tested?

srowen Feb 20, 2019

Choose a reason for hiding this comment

SparkQA commented Feb 22, 2019

srowen commented Feb 25, 2019

SparkQA commented Feb 26, 2019

liupc commented Mar 1, 2019

liupc commented Mar 1, 2019

SparkQA commented Mar 1, 2019

liupc commented Mar 5, 2019

liupc commented Mar 6, 2019

liupc commented Mar 6, 2019

SparkQA commented Mar 6, 2019

SparkQA commented Mar 9, 2019

SparkQA commented Mar 11, 2019

srowen commented Mar 11, 2019 • edited

SparkQA commented Mar 11, 2019

SparkQA commented Mar 12, 2019

vanzin left a comment

Choose a reason for hiding this comment

vanzin Mar 13, 2019

Choose a reason for hiding this comment

liupc Mar 13, 2019

Choose a reason for hiding this comment

vanzin commented Mar 13, 2019

SparkQA commented Mar 13, 2019

SparkQA commented Mar 15, 2019

srowen commented Mar 15, 2019

SparkQA commented Mar 16, 2019

srowen commented Mar 17, 2019

srowen commented Mar 11, 2019 •

edited