[SPARK-6018] [YARN] NoSuchMethodError in Spark app is swallowed by YARN AM #4773
Conversation
|
Can one of the admins verify this patch? |
| // Reporter thread can interrupt to stop user class | ||
| case e: Exception => | ||
| case _ => |
There was a problem hiding this comment.
So, the problem is that this was not catching Error, but just Exception, right?
I think this generates a compiler warning, so probably should be case _: Throwable =>. And you can avoid the duplicate getCause calls by giving this variable a name instead of using the placeholder.
| @@ -483,14 +483,14 @@ private[spark] class ApplicationMaster( | |||
| } catch { | |||
| case e: InvocationTargetException => | |||
| e.getCause match { | |||
| case _: InterruptedException => | |||
| case e: InterruptedException => | |||
There was a problem hiding this comment.
Nit: reusing the name e is ambiguous but also unnecessary; this can be _ too? (Yes the second clause will generate a warning without : Throwable
|
Thank you Marcelo and Sean. Incorporated your comments in the new commit. |
| @@ -485,7 +485,7 @@ private[spark] class ApplicationMaster( | |||
| e.getCause match { | |||
| case _: InterruptedException => | |||
| // Reporter thread can interrupt to stop user class | |||
| case e: Exception => | |||
| case e: Throwable => | |||
There was a problem hiding this comment.
e still shadows the e: InvocationTargetException above. I know it was already that way in the code and it works as intended this way. However while we're here I wonder if it's worth calling this variable cause? then it's obvious that cause is being rethrown.
There was a problem hiding this comment.
Sure. That is even better. Fixed.
|
ok to test |
|
Test build #27967 has started for PR 4773 at commit
|
|
Test build #27967 has finished for PR 4773 at commit
|
|
Test PASSed. |
|
LGTM. Let me leave it open for a day at least for other thoughts. |
|
Ah, good catch. For a second I thought we were catching throwable here, but it seems that the throwable is part of the cause, not what we're catching. LGTM I'm merging this into master 1.3 and 1.2. |
…RN AM Author: Cheolsoo Park <cheolsoop@netflix.com> Closes #4773 from piaozhexiu/SPARK-6018 and squashes the following commits: 2a919d5 [Cheolsoo Park] Rename e with cause to avoid duplicate names 1e71d2d [Cheolsoo Park] Replace placeholder with throwable eb5750d [Cheolsoo Park] NoSuchMethodError in Spark app is swallowed by YARN AM (cherry picked from commit 5f3238b) Signed-off-by: Andrew Or <andrew@databricks.com>
…RN AM Author: Cheolsoo Park <cheolsoop@netflix.com> Closes #4773 from piaozhexiu/SPARK-6018 and squashes the following commits: 2a919d5 [Cheolsoo Park] Rename e with cause to avoid duplicate names 1e71d2d [Cheolsoo Park] Replace placeholder with throwable eb5750d [Cheolsoo Park] NoSuchMethodError in Spark app is swallowed by YARN AM (cherry picked from commit 5f3238b) Signed-off-by: Andrew Or <andrew@databricks.com>
No description provided.