[SPARK-12265][Mesos] Spark calls System.exit inside driver instead of throwing exception

[dragos]

This takes over #10729 and makes sure that spark-shell fails with a proper error message. There is a slight behavioral change: before this change spark-shell would exit, while now the REPL is still there, but sc and sqlContext are not defined and the error is visible to the user.

[SparkQA]

Test build #50094 has finished for PR 10921 at commit 639ef36.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[skyluc]

Tested with spark-shell, spark-submit in client and cluster mode, and embedded in an application.
It worked, throws the exception but doesn't kill the application.
LGTM

[nraychaudhuri]

Thanks for taking this up @dragos

[andrewor14]

LGTM merging into master.

[IgorBerman]

@dragos @andrewor14 @srowen I believe I have some corner case connected to this fix:
Suppose I have spark driver embedded in java service and mesosDriver.run() exits for any reason(in my case due to temporary un-availability of external shuffle service on one of the worker nodes)

currently there is no way to get this information besides checking if "-mesos-driver" exists somehow. I.e. the initialisation sequence went smoothly but due to dynamic allocation quirks mesos-driver exits(so countDown latch will be released already at this point)

I'm wondering if I can get this signal with some spark listener?
Should I create additional jira for this?

WDYT?