[SPARK-11801][CORE] Notify driver when OOM is thrown before executor …

[vundela]

…JVM is killed

This fix try to make sure that task which caught OOM will update its status to driver so that driver logs will have enough information why the tasks are lost or executor is lost. This fix does the following

1. Registers a shutdown hook for executor which does the following
   a) Synchronizes with OOM handler thread (Assumption is that OOM thread is still running and gets the lock prior to the shutdown hook thread. I thought of introducing some delay, but my runs with fix several times didn't get to that situation.)
   b) Kill all the remaining tasks running in the current container( I thought it would be good to clean the task properly, so that they wont do any job which might throw unwanted error/exceptions)
   c) Sleeps some time so that OOM handler status is flushed to driver(No sleeping causes the status message lost)

2. Separate handler for OOM, so that we can send proper message to driver.

[vundela]

Here is snippet of the messages in driver logs

15/11/19 16:31:23 INFO YarnAllocator: Canceling requests for 1 executor containers
15/11/19 16:31:23 WARN TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6, vsr-4.vpc.cloudera.com): TaskOutOfMemory (task caught OutOfMemoryError)
15/11/19 16:31:23 INFO TaskSetManager: Starting task 6.1 in stage 0.0 (TID 14, vsr-4.vpc.cloudera.com, partition 6,PROCESS_LOCAL, 1983 bytes)
15/11/19 16:31:23 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 15, vsr-4.vpc.cloudera.com, partition 14,PROCESS_LOCAL, 1987 bytes)
15/11/19 16:31:23 WARN TaskSetManager: Lost task 7.0 in stage 0.0 (TID 7, vsr-4.vpc.cloudera.com): TaskKilled (killed intentionally)
15/11/19 16:31:24 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. vsr-4.vpc.cloudera.com:41617
15/11/19 16:31:24 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. vsr-4.vpc.cloudera.com:41617
15/11/19 16:31:24 ERROR YarnClusterScheduler: Lost executor 2 on vsr-4.vpc.cloudera.com: remote Rpc client disassociated

[squito]

JavaConversions is discouraged (its hard for the reader to see where the conversion is happening). stick to JavaConverters and explicitly call asScala / asJava

[squito]

Hi @vundela, I left a few comments on the code, but there are actually some larger design issues which I think need to be discussed first. I'd like to move that discussion to the jira (so its better archived) before moving forward with this further.

[AmplabJenkins]

Can one of the admins verify this patch?