Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown #28746

Closed
wants to merge 8 commits into from
Closed

[SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown #28746

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d2555c2
fix
Ngone51 Jun 7, 2020
db435d2
fix
Ngone51 Jun 8, 2020
7a713ac
swith the line
Ngone51 Jul 9, 2020
09bbb03
revert the change
Ngone51 Jul 10, 2020
092c442
use debug
Ngone51 Jul 10, 2020
ef576c0
exchange
Ngone51 Jul 10, 2020
4c8ceec
add comment
Ngone51 Jul 10, 2020
59a0595
no spacrs
Ngone51 Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
View file
Open in desktop
Expand Up @@ -76,8 +76,8 @@ class LocalSparkCluster(
logInfo("Shutting down local Spark cluster.")
// Stop the workers before the master so they don't get upset that it disconnected
workerRpcEnvs.foreach(_.shutdown())
Ngone51 marked this conversation as resolved.
Show resolved Hide resolved
masterRpcEnvs.foreach(_.shutdown())
workerRpcEnvs.foreach(_.awaitTermination())
Ngone51 marked this conversation as resolved.
Show resolved Hide resolved
masterRpcEnvs.foreach(_.shutdown())
masterRpcEnvs.foreach(_.awaitTermination())
masterRpcEnvs.clear()
workerRpcEnvs.clear()
Expand Down
9 changes: 8 additions & 1 deletion core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
View file
Open in desktop
Expand Up @@ -147,7 +147,14 @@ private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) exte
/** Posts a one-way message. */
def postOneWayMessage(message: RequestMessage): Unit = {
postMessage(message.receiver.name, OneWayMessage(message.senderAddress, message.content),
(e) => throw e)
(e) => e match {
// SPARK-31922: in local cluster mode, there's always a RpcEnvStoppedException when
// stop is called due to some asynchronous message handling. We catch the exception
// and log it at debug level to avoid verbose error message when user stop a local
// cluster in spark shell.
case re: RpcEnvStoppedException => logDebug (s"Message $message dropped. ${re.getMessage}")
Ngone51 marked this conversation as resolved.
Show resolved Hide resolved
case _ => throw e
})
}

/**
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
View file
Open in desktop
Expand Up @@ -560,7 +560,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
}

override def reviveOffers(): Unit = {
override def reviveOffers(): Unit = Utils.tryLogNonFatalError {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change fixes the failure of test org.apache.spark.launcher.LauncherBackendSuite.standalone/client: launcher handle. After sleeping one more second, the application launched by the SparkLauncher now has a chance to submit tasks to TaskScheduler and call reviveOffers on the SchedulerBackend. At the same time, the SparkLauncher will ask the application to stop. Therefore, the SchedulerBackend could have been already stopped when it receives ReviveOffers messages, which would fail the entire application at the end.

So, I use Utils.tryLogNonFatalError to fix it and I think this should be fine since we've already use it at:

spark/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala

Lines 137 to 139 in c560428

reviveThread.scheduleAtFixedRate(() => Utils.tryLogNonFatalError {
Option(self).foreach(_.send(ReviveOffers))
}, 0, reviveIntervalMs, TimeUnit.MILLISECONDS)

driverEndpoint.send(ReviveOffers)
}

Expand Down