[SPARK-6443][Spark Submit]Could not submit app in standalone cluster mode when HA is enabled #5116

WangTaoTheTonic · 2015-03-21T08:46:55Z

3/26 update:

Akka-based:
Use an array of ActorSelection to represent multiple master. Add an activeMasterActor for query status of driver. And will add lost masters( including the standby one) to lostMasters.
When size of lostMasters equals or greater than # of all masters, we should give an error that all masters are not avalible.
Rest-based:
When all masters are not available(throw an exception), we use akka gateway to submit apps.

I have tested simply on standalone HA cluster(with two masters alive and one alive/one dead), it worked.

There might remains some issues on style or message print, but we can check the solution then fix them together.

/cc @srowen @andrewor14

SparkQA · 2015-03-21T09:20:36Z

Test build #28942 has finished for PR 5116 at commit da4ad2e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SubmitDriverResponse(
- case class KillDriverResponse(

andrewor14 · 2015-03-25T02:07:31Z

retest this please

SparkQA · 2015-03-25T03:11:53Z

Test build #29138 has finished for PR 5116 at commit da4ad2e.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- case class SubmitDriverResponse(
- case class KillDriverResponse(

SparkQA · 2015-03-25T05:05:56Z

Test build #29147 has finished for PR 5116 at commit 60d97a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-03-25T05:17:48Z

Test build #29146 has finished for PR 5116 at commit fa1fa80.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2015-03-25T09:24:17Z

Test build #29156 has finished for PR 5116 at commit 2b011c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-03-25T14:45:33Z

Though passed all the tests, it still doesn't work well in some cases.
I'm still working on it.

SparkQA · 2015-03-26T04:51:48Z

Test build #29205 has finished for PR 5116 at commit 7a881b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-03-26T07:17:19Z

The UT failed because of hive compile error which would be fixed in #5198.

WangTaoTheTonic · 2015-03-26T09:10:29Z

Jenkins, test this please.

SparkQA · 2015-03-26T11:13:11Z

Test build #29222 timed out for PR 5116 at commit 7a881b3 after a configured wait of 120m.

WangTaoTheTonic · 2015-03-26T12:59:09Z

Jenkins, test this please.

SparkQA · 2015-03-26T14:07:11Z

Test build #29229 has finished for PR 5116 at commit 7a881b3.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-03-26T14:22:01Z

Jenkins, test this please.

SparkQA · 2015-03-26T15:44:14Z

Test build #29235 has finished for PR 5116 at commit 7a881b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-04-01T02:29:48Z

@andrewor14 Could you please take a look?

andrewor14 · 2015-04-02T22:01:20Z

core/src/main/scala/org/apache/spark/deploy/Client.scala

+      if (success) {
+        activeMasterActor = context.actorSelection(sender.path)
+        pollAndReportStatus(driverId.get)
+      } else if (!message.contains("Can only")) {


can you explain what "Can only" is?

Ah, I see, it's from this message: Can only accept driver submissions in ALIVE state.

WangTaoTheTonic · 2015-04-16T10:56:02Z

Jenkins, test this please.

SparkQA · 2015-04-16T12:27:01Z

Test build #30416 has finished for PR 5116 at commit 220cb3c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-22T01:35:26Z

Test build #30719 has finished for PR 5116 at commit a41de0b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-22T04:57:14Z

Test build #30724 has finished for PR 5116 at commit f4f972b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

WangTaoTheTonic · 2015-04-22T06:24:01Z

@andrewor14 I've changed and made some rebase. If everything is ok i'd like to merge it ASAP. Please take a look.

WangTaoTheTonic · 2015-04-26T12:58:19Z

ping @andrewor14

SparkQA · 2015-04-27T18:21:11Z

Test build #31047 has started for PR 5116 at commit f4f972b.

WangTaoTheTonic · 2015-04-28T01:35:44Z

Looks like Jenkins has done his work but not posted the result here.

andrewor14 · 2015-04-28T19:41:37Z

retest this please

andrewor14 · 2015-04-28T19:46:58Z

I think catching SubmitRestConnectionException in createSubmission/requestSubmissionStatus/killSubmission is more reasonable than in readResponse as we use requestSubmissionStatus to Read the response from the server and return it as a validated and we submit requests to multiple masters in createSubmission/requestSubmissionStatus/killSubmission so we should take the response there either.

If you look into the these methods you'll notice that the only place where it actually makes a connection to the master is in post or postJson, and both of these methods call readResponse. The current scope of catching these exceptions is too wide (I believe we will never throw this exception outside of readResponse), and the cost we pay is duplicate code.

andrewor14 · 2015-04-28T19:47:32Z

By the way the rest of the changes look fine to me.

SparkQA · 2015-04-28T21:10:53Z

Test build #31161 has finished for PR 5116 at commit f4f972b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

WangTaoTheTonic · 2015-04-29T02:16:52Z

I believe we will never throw this exception outside of readResponse

@andrewor14 No. In postJson, it will throw a ConnectException when try to get output stream of a connection while the remote server is down. I take care of it here:

try {
val out = new DataOutputStream(conn.getOutputStream)
Utils.tryWithSafeFinally {
out.write(json.getBytes(Charsets.UTF_8))
} {
out.close()
}
} catch {
case e: ConnectException =>
throw new SubmitRestConnectionException("Connect Exception when connect to server", e)
}

andrewor14 · 2015-04-29T02:57:00Z

I see, that seems to be the case, then I would do it in get, post and postJson instead of the higher level methods.

WangTaoTheTonic · 2015-04-29T03:16:15Z

Sorry but I don't see benefit to push it down which still have same duplicated code.

I think we should catch SubmitRestConnectionException while send an request to one master rather than post a request to some url. Code-wise they are same in this case, the meaning is different though.

SparkQA · 2015-04-29T04:20:43Z

Test build #31227 has finished for PR 5116 at commit 76fd411.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-29T08:40:29Z

Test build #31259 has finished for PR 5116 at commit 2a28aab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

andrewor14 · 2015-04-29T18:26:39Z

Ok, no problem. The slightly nice thing is that it hides the details from the higher level and only handle it in relatively low level methods (e.g. post), but not a big deal. I will merge this first and we can always do any clean ups later if needed. I don't want to make you keep rebasing.

andrewor14 · 2015-04-29T18:26:49Z

But first, retest this please

SparkQA · 2015-04-29T20:01:56Z

Test build #31303 has finished for PR 5116 at commit 2a28aab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

WangTaoTheTonic · 2015-04-30T01:46:13Z

@andrewor14 Thanks for all comments. :)

andrewor14 · 2015-05-02T01:37:07Z

Merging into master.

…r mode when HA is enabled **3/26 update:** * Akka-based: Use an array of `ActorSelection` to represent multiple master. Add an `activeMasterActor` for query status of driver. And will add lost masters( including the standby one) to `lostMasters`. When size of `lostMasters` equals or greater than # of all masters, we should give an error that all masters are not avalible. * Rest-based: When all masters are not available(throw an exception), we use akka gateway to submit apps. I have tested simply on standalone HA cluster(with two masters alive and one alive/one dead), it worked. There might remains some issues on style or message print, but we can check the solution then fix them together. /cc srowen andrewor14 Author: WangTaoTheTonic <wangtao111@huawei.com> Closes apache#5116 from WangTaoTheTonic/SPARK-6443 and squashes the following commits: 2a28aab [WangTaoTheTonic] based the newest change apache#5144 76fd411 [WangTaoTheTonic] rebase f4f972b [WangTaoTheTonic] rebase...again a41de0b [WangTaoTheTonic] rebase 220cb3c [WangTaoTheTonic] move connect exception inside 35119a0 [WangTaoTheTonic] style and compile issues 9d636be [WangTaoTheTonic] per Andrew's comments 979760c [WangTaoTheTonic] rebase e4f4ece [WangTaoTheTonic] fix failed test 5d23958 [WangTaoTheTonic] refact some duplicated code, style and comments 7a881b3 [WangTaoTheTonic] when one of masters is gone, we still can submit 2b011c9 [WangTaoTheTonic] fix broken tests 60d97a4 [WangTaoTheTonic] rebase fa1fa80 [WangTaoTheTonic] submit app to HA cluster in standalone cluster mode

submit app to HA cluster in standalone cluster mode

fa1fa80

WangTaoTheTonic force-pushed the SPARK-6443 branch from da4ad2e to fa1fa80 Compare March 25, 2015 03:48

rebase

60d97a4

fix broken tests

2b011c9

when one of masters is gone, we still can submit

7a881b3

WangTaoTheTonic changed the title ~~[SPARK-6443][WIP]Could not submit app in standalone cluster mode when HA is enabled~~ [SPARK-6443][Spark Submit]Could not submit app in standalone cluster mode when HA is enabled Mar 26, 2015

andrewor14 reviewed Apr 2, 2015
View reviewed changes

rebase

a41de0b

rebase...again

f4f972b

rebase

76fd411

based the newest change apache#5144

2a28aab

asfgit closed this in b4b43df May 2, 2015

[SPARK-6443][Spark Submit]Could not submit app in standalone cluster mode when HA is enabled #5116

[SPARK-6443][Spark Submit]Could not submit app in standalone cluster mode when HA is enabled #5116

Conversation

WangTaoTheTonic commented Mar 21, 2015

SparkQA commented Mar 21, 2015

andrewor14 commented Mar 25, 2015

SparkQA commented Mar 25, 2015

SparkQA commented Mar 25, 2015

SparkQA commented Mar 25, 2015

SparkQA commented Mar 25, 2015

WangTaoTheTonic commented Mar 25, 2015

SparkQA commented Mar 26, 2015

WangTaoTheTonic commented Mar 26, 2015

WangTaoTheTonic commented Mar 26, 2015

SparkQA commented Mar 26, 2015

WangTaoTheTonic commented Mar 26, 2015

SparkQA commented Mar 26, 2015

WangTaoTheTonic commented Mar 26, 2015

SparkQA commented Mar 26, 2015

WangTaoTheTonic commented Apr 1, 2015

andrewor14 Apr 2, 2015

Choose a reason for hiding this comment

andrewor14 Apr 2, 2015

Choose a reason for hiding this comment

WangTaoTheTonic commented Apr 16, 2015

SparkQA commented Apr 16, 2015

SparkQA commented Apr 22, 2015

SparkQA commented Apr 22, 2015

WangTaoTheTonic commented Apr 22, 2015

WangTaoTheTonic commented Apr 26, 2015

SparkQA commented Apr 27, 2015

WangTaoTheTonic commented Apr 28, 2015

andrewor14 commented Apr 28, 2015

andrewor14 commented Apr 28, 2015

andrewor14 commented Apr 28, 2015

SparkQA commented Apr 28, 2015

WangTaoTheTonic commented Apr 29, 2015

andrewor14 commented Apr 29, 2015

WangTaoTheTonic commented Apr 29, 2015

SparkQA commented Apr 29, 2015

SparkQA commented Apr 29, 2015

andrewor14 commented Apr 29, 2015

andrewor14 commented Apr 29, 2015

SparkQA commented Apr 29, 2015

WangTaoTheTonic commented Apr 30, 2015

andrewor14 commented May 2, 2015