[SPARK-16868][Web Ui] Fix executor be both dead and alive on executor ui.

[SaintBacchus]

What changes were proposed in this pull request?

In a heavy pressure of the spark application, since the executor will register it to driver block manager twice(because of heart beats), the executor will show as picture show:

How was this patch tested?

NA

Details in: SPARK-16868

[SparkQA]

Test build #63335 has finished for PR 14530 at commit 85b385f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class ShuffleIndexInformation
  • public class ShuffleIndexRecord

[srowen]

Just commenting on the code, not the logic: isn't this just something like deadExecutorStorageStatus.retain(_.blockManagerId.executorId != executorId)? I know it does something slightly different, but maybe it's desirable to clear all such matching statuses. And it's a one-liner that doesn't need a method.

[SaintBacchus]

retain seems to be a method in MapLike, but I can't find any similar method in ListBuffer.

[srowen]

Ah. OK what about

val toRemove = deadExecutorStorageStatus.find(_.blockManagerId.executorId == executorId)
if (toRemove >= 0) {
  deadExecutorStorageStatus.remove(toRemove)
}

Might be clearer and more direct. No big deal either way.

[SparkQA]

Test build #63355 has finished for PR 14530 at commit c065661.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SaintBacchus]

\cc @srowen

[srowen]

As to the actual logic of the change, I don't feel that confident about reviewing it. Suggest looking at 'blame' output here to see who might have written it?

[vanzin]

Looks fine (small style nit but not worth another review round). I'm a little curious about why it happens - I thought that the driver would kill executors that failed a heartbeat, but that's a separate discussion.

Merging to master.

[SaintBacchus]

I will re-run this case, and dig into why the executor will double register.