[SPARK-8560][UI] The Executors page will have negative if having resubmitted tasks

[XuTingjun]

when the taskEnd.reason is Resubmitted, it shouldn't do statistics. Because this tasks has a SUCCESS taskEnd before.

[SparkQA]

Test build #35538 has finished for PR 6950 at commit af35dc3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

This looks fine to me. I'll leave it a little in case others have more to add.

[XuTingjun]

@tdas @pwendell Can you have a look on this patch, thanks!

[tdas]

@JoshRosen is a better person to check this out.

[andrewor14]

don't we still want to update the shuffle metrics down there?

[XuTingjun]

I think so, because these Resubmitted SparkListenerTaskEnd tasks have post a Success SparkListenerTaskEnd before, all the metrics have been updated. If update here, the metrics will calculate twice.

[andrewor14]

hm, technically this is still incorrect because it uses the metrics of the first attempt of this task, which might be incomplete. However, a correct fix would be much more complicated (you have to essentially revert the metrics added in the failed task).

We should at least add a comment:

// Note: For resubmitted tasks, we continue to use the metrics that belong to the
// first attempt of this task. This may not be 100% accurate because the first attempt
// could have failed half-way through. The correct fix would be to keep track of the
// metrics added by each attempt, but this is much more complicated.

[andrewor14]

Alright, this is good to go after you add that comment. Thanks @XuTingjun

[andrewor14]

retest this please

[SparkQA]

Test build #36166 has finished for PR 6950 at commit af35dc3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Alright, I fixed the comment myself and merged this into master. Thanks @XuTingjun.