[SPARK-12339] [WebUI] Added a null check that was removed in SPARK-11206

[ajbozarth]

Updates made in SPARK-11206 missed an edge case which cause's a NullPointerException when a task is killed. In some cases when a task ends in failure taskMetrics is initialized as null (see JobProgressListener.onTaskEnd()). To address this a null check was added. Before the changes in SPARK-11206 this null check was called at the start of the updateTaskAccumulatorValues() function.

[ajbozarth]

Looping in those involved with SPARK-11206: @carsonwang @JoshRosen @vanzin

[ajbozarth]

FYI the line in JobProgressListener.onTaskEnd that initializes the null value is 387.

[SparkQA]

Test build #48089 has finished for PR 10405 at commit 0a46559.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[carsonwang]

Thanks for catching this. I think the null check here is necessary, and it seems the code that really pass a null taskMetrcis is from the TaskSetManager line 796 when a task is resubmitted because of executor lost.

[srowen]

Maybe a dumb question, but is it right that calling updateTaskAccumulatorValues is meaningless if there are no taskMetrics? that is, should the call not happen at all or should a different arg be passed rather than the arg that causes an NPE? otherwise LGTM

[ajbozarth]

@srowen I thought the same thing so I looked into git history and git blame and found that this is how updateTaskAccumulatorValues has always worked, it's original code up until the change in SPARK-11206 had a null check that immediately returned at the start of the function.

[andrewor14]

OK, merging into master, thanks.

[Naresh523]

I'm seeing this issue been fixed in v2.0.0. Is there any possibility to fix this in v1.6.x ?

[srowen]

@Naresh523 this particular issue doesn't appear to exist in 1.6 because the change that this fixes went into 2.0

[Naresh523]

I'm getting similar error logs [1]. there were no changes in application.
I'm using spark v1.6.0

[1] https://issues.apache.org/jira/browse/SPARK-12339

[VishnuGowthemT]

Can this fix be added in 1.6 as well ?
https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala