[SPARK-8754][YARN] YarnClientSchedulerBackend doesn't stop gracefully in failure conditions

[devaraj-kavali]

In YarnClientSchedulerBackend.stop(), added a check for monitorThread.

[srowen]

LGTM

[SparkQA]

Test build #991 has finished for PR 7153 at commit 66be9ad.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Heartbeat(workerId: String, worker: RpcEndpointRef) extends DeployMessage
  • case class RegisteredWorker(master: RpcEndpointRef, masterWebUiUrl: String) extends DeployMessage
  • case class RegisterApplication(appDescription: ApplicationDescription, driver: RpcEndpointRef)
  • case class RegisteredApplication(appId: String, master: RpcEndpointRef) extends DeployMessage
  • case class SubmitDriverResponse(
  • case class KillDriverResponse(
  • case class MasterChanged(master: RpcEndpointRef, masterWebUiUrl: String)
  • class DCT(override val uid: String)
  • class MinMaxScaler(override val uid: String)
  • class PCA (override val uid: String) extends Estimator[PCAModel] with PCAParams
  • class StreamingLinearAlgorithm(object):
  • class StreamingLinearRegressionWithSGD(StreamingLinearAlgorithm):
  • class AnalysisException(Exception):
  • case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
  • class SpecificOrdering extends $
  • class SpecificProjection extends $
  • final class SpecificRow extends $
  • case class Crc32(child: Expression)
  • // compiled class file for the closure here will conflict with the one in callUDF (upper case).

[squito]

jenkins, retest this please

[andrewor14]

retest this please

[SparkQA]

Test build #36317 has finished for PR 7153 at commit 66be9ad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Merging into master 1.4. Thanks @devaraj-kavali