[SPARK-12967][Netty] Avoid NettyRpc error message during sparkContext shutdown #10881

nishkamravi2 · 2016-01-22T22:47:03Z

If there's an RPC issue while sparkContext is alive but stopped (which would happen only when executing SparkContext.stop), log a warning instead. This is a common occurrence.

@vanzin

SparkQA · 2016-01-23T00:47:20Z

Test build #49924 has finished for PR 10881 at commit 2c07186.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-23T01:31:20Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -2183,6 +2183,13 @@ object SparkContext extends Logging {
  }

  /**
+   * Return SparkContext
+   */
+  def get(): SparkContext = {


Is this meant to be part of the public API or should this be private?

@holdenk, figured this may be a good utility to have in SparkContext alongside getOrCreate. Unless you think otherwise?

No please avoid introducing new random APIs in an unrelated pull request.

Also it would be great to not have the lower level depend on an upper level (e.g. network rpc depending on sparkcontext).

@rxin Modified it to private. Any ideas on how to fix this without using SparkContext?

We could potentially just drop a message if it cannot be sent and log a warning?

rxin · 2016-01-23T07:58:15Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -2183,6 +2183,13 @@ object SparkContext extends Logging {
  }

  /**
+   * Return SparkContext
+   */
+  private[spark] def get(): SparkContext = {


I think it's best to just remove this get method. We should have the upper layer (i.e. sparkcontext) send an explicit signal via function calls to the lower layer (e.g. rpc).

@zsxwing might know more about whether this is needed at all.

Actually, there is already a such explicit signal: RpcEnv.shutdown. Previously, I was worried about the immature NettyRpcEnv. So the stack trace of the error is still logged to help debug after RpcEnv is shutdown. However, since NettyRpcEnv becomes mature, we can remove the stack trace.

zsxwing · 2016-01-23T22:24:25Z

@nishkamravi2 I suggest that you add a new exception RpcEnvStopException that extends IllegalStateException. Then you can use RpcEnvStopException in this line:

spark/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala

Line 159 in 7a375bb

new IllegalStateException("RpcEnv already stopped.")

After that, there are two places need to handle this exception:

Try-catch RpcEnvStopException in the following line:

spark/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala

Line 185 in 7a375bb

dispatcher.postOneWayMessage(message)

and only log a simple message
2. Check the exception type in the following line:

spark/core/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala

Line 46 in 7a375bb

logWarning(s"Failed to send one-way RPC.", e)

If it's RpcEnvStopException, log a simple message.

nishkamravi2 · 2016-01-23T22:35:03Z

Thanks for the pointers @zsxwing. Makes sense to just log an error message. Instead of throwing an exception from postMessage and logging it in send, what if we just log the warning in postMessage directly?

zsxwing · 2016-01-23T22:43:05Z

Thanks for the pointers @zsxwing. Makes sense to just log an error message. Instead of throwing an exception from postMessage and logging it in send, what if we just log the warning in postMessage directly?

If this is called from ask, we still need to throw the exception so that we can notify the caller. The user may use something like Await.result(endpointRef.ask(...)).

nishkamravi2 · 2016-01-23T23:13:32Z

Ok, makes sense. Btw, we also want to mask the exception being thrown at line 161 (I see that in the logs as well).

…Exception)

nishkamravi2 · 2016-01-25T05:55:13Z

@zsxwing Updated the PR after some testing. Most of the stack trace in the log was coming from postToAll RPC for RemoteProcessDisconnected, so I have removed the stack trace from the callback.

SparkQA · 2016-01-25T07:40:34Z

Test build #49979 has finished for PR 10881 at commit 3376bfb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-01-26T21:53:44Z

LGTM.

zsxwing · 2016-01-26T22:02:29Z

core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala

@@ -106,7 +106,7 @@ private[netty] class Dispatcher(nettyEnv: NettyRpcEnv) extends Logging {
    val iter = endpoints.keySet().iterator()
    while (iter.hasNext) {
      val name = iter.next
-      postMessage(name, message, (e) => logWarning(s"Message $message dropped.", e))
+      postMessage(name, message, (e) => logWarning(s"Message $message dropped."))


Could you update the message as s"Message $message dropped. ${e.getMessage}"? It's worth to add the exception message here since there are two types of exceptions.

zsxwing · 2016-01-26T22:03:26Z

Looks pretty good. Will merge it once you update the warning message.

nishkamravi2 · 2016-01-27T03:13:20Z

Updated the warning message. Thanks.

SparkQA · 2016-01-27T05:08:37Z

Test build #50166 has finished for PR 10881 at commit f91afed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-27T05:12:49Z

@nishkamravi2 thanks, merging to master

…ntext shutdown If there's an RPC issue while sparkContext is alive but stopped (which would happen only when executing SparkContext.stop), log a warning instead. This is a common occurrence. vanzin Author: Nishkam Ravi <nishkamravi@gmail.com> Author: nishkamravi2 <nishkamravi@gmail.com> Closes apache#10881 from nishkamravi2/master_netty. (cherry picked from commit bae3c9a)

JerryLead · 2016-07-19T02:59:31Z

This bug still exists in latest Spark 1.6.2. How about merging it to branch-1.6? @nishkamravi2 @zsxwing

1236897 · 2016-10-21T13:43:34Z

could i know how to merge the updated code to my project to avoid this error?

zsxwing · 2016-10-21T20:24:34Z

@1236897 Just try git cherry-pick this PR.

1236897 · 2016-11-03T09:51:32Z

actually, I face the issue that "RpcEnv already stopped" when i call sparkContxt.stop in the end of the program. I add the spark1.6 through maven pom. could i know how to fix this issue ?

zsxwing · 2016-11-03T17:50:40Z

@1236897 You can check out Spark 1.6.2 tag and apply this patch. Then build the Spark and use this one to submit your Spark application. You can still use the Spark maven artifact to build your application.

1236897 · 2016-11-03T18:01:41Z

@zsxwing Thank you for your reply and sorry to disturb you, cos this project is so import for me, I descrise what i need to do. Firistly, check out the spark 1.6 from Github. Secondly, use the git cherry-pick to apply this patch and compile the code through ecplise. Lastly, add the output jar to maven repository and add this jar as dependency to my project. right ?

zsxwing · 2016-11-03T18:12:51Z

compile the code through ecplise.

No. Take a look at this page about how to build Spark: http://spark.apache.org/docs/1.6.2/building-spark.html

Lastly, add the output jar to maven repository and add this jar as dependency to my project.

You don't need to. You can still use the official maven artifact to build your application. However, you need to use the new spark-submit inside your own Spark build to submit your application. If you use the standalone Spark cluster and run the Spark app using the cluster mode, you must upgrade your Spark cluster as well.

zsxwing · 2016-11-03T18:15:07Z

@1236897 If you don't want to build Spark, it's fine to just catch this special exception thrown from SparkContxt.stop and ignore it.

1236897 · 2016-11-03T18:21:42Z

@ zsxwing if I just ignore it, will it avoid my issue? my issue is I need complete my job within 5 mins. but the issue "RpcEnv already stopped" waste a lot of time to disconnect and make my job beyond my expect.

zsxwing · 2016-11-03T18:50:48Z

the issue "RpcEnv already stopped" waste a lot of time to disconnect

Did you measure how long to stop SparkContext? I don't think it will take several minutes.

1236897 · 2016-11-03T19:09:33Z

cos I watch the monitoring page. my last step is save as parquet. but it page monitoring page stage show completed and the job keep running, after few minutes, the job is complete and throw the Expention about "RpcEnv already stopped" , that's the reason I assume the Excetpion lead to the long RUNNING and not Complete. PS: the job use 40 executors in the 200 servers Env.

zsxwing · 2016-11-03T19:22:44Z

@1236897 The Spark job status from RUNNING to COMPLETE happens before SparkContext.stop.

fan1emon · 2017-03-09T09:46:37Z

HI, @zsxwing. Is there any way to reproduce this issue?

Avoid Netty RPC error messages when sparkContext stops

2c07186

holdenk reviewed Jan 23, 2016
View reviewed changes

Update SparkContext.scala

b61ac28

rxin reviewed Jan 23, 2016
View reviewed changes

Update (remove stack trace from postToAll callback, add RpcEnvStopped…

3376bfb

…Exception)

vanzin mentioned this pull request Jan 25, 2016

[SPARK-12614][Core]Don't throw non fatal exception from ask #10568

Closed

zsxwing reviewed Jan 26, 2016
View reviewed changes

Update Dispatcher.scala

f91afed

asfgit closed this in bae3c9a Jan 27, 2016

jiacai2050 mentioned this pull request Nov 22, 2016

java.lang.IllegalStateException: RpcEnv already stopped. yahoo/CaffeOnSpark#83

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12967][Netty] Avoid NettyRpc error message during sparkContext shutdown #10881

[SPARK-12967][Netty] Avoid NettyRpc error message during sparkContext shutdown #10881

nishkamravi2 commented Jan 22, 2016

SparkQA commented Jan 23, 2016

holdenk Jan 23, 2016

nishkamravi2 Jan 23, 2016

rxin Jan 23, 2016

nishkamravi2 Jan 23, 2016

nishkamravi2 Jan 23, 2016

rxin Jan 23, 2016

zsxwing Jan 23, 2016

zsxwing commented Jan 23, 2016

nishkamravi2 commented Jan 23, 2016

zsxwing commented Jan 23, 2016

nishkamravi2 commented Jan 23, 2016

nishkamravi2 commented Jan 25, 2016

SparkQA commented Jan 25, 2016

vanzin commented Jan 26, 2016

zsxwing Jan 26, 2016

zsxwing commented Jan 26, 2016

nishkamravi2 commented Jan 27, 2016

SparkQA commented Jan 27, 2016

zsxwing commented Jan 27, 2016

JerryLead commented Jul 19, 2016

1236897 commented Oct 21, 2016

zsxwing commented Oct 21, 2016 •

edited

Loading

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

fan1emon commented Mar 9, 2017 •

edited

Loading

[SPARK-12967][Netty] Avoid NettyRpc error message during sparkContext shutdown #10881

[SPARK-12967][Netty] Avoid NettyRpc error message during sparkContext shutdown #10881

Conversation

nishkamravi2 commented Jan 22, 2016

SparkQA commented Jan 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zsxwing commented Jan 23, 2016

nishkamravi2 commented Jan 23, 2016

zsxwing commented Jan 23, 2016

nishkamravi2 commented Jan 23, 2016

nishkamravi2 commented Jan 25, 2016

SparkQA commented Jan 25, 2016

vanzin commented Jan 26, 2016

Choose a reason for hiding this comment

zsxwing commented Jan 26, 2016

nishkamravi2 commented Jan 27, 2016

SparkQA commented Jan 27, 2016

zsxwing commented Jan 27, 2016

JerryLead commented Jul 19, 2016

1236897 commented Oct 21, 2016

zsxwing commented Oct 21, 2016 • edited Loading

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

1236897 commented Nov 3, 2016

zsxwing commented Nov 3, 2016

fan1emon commented Mar 9, 2017 • edited Loading

zsxwing commented Oct 21, 2016 •

edited

Loading

fan1emon commented Mar 9, 2017 •

edited

Loading