[SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager etc. #2138

sarutak · 2014-08-26T15:30:37Z

No description provided.

SparkQA · 2014-08-26T15:35:54Z

QA tests have started for PR 2138 at commit e5ad9d3.

This patch merges cleanly.

SparkQA · 2014-08-26T16:13:26Z

QA tests have finished for PR 2138 at commit e5ad9d3.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-26T16:40:53Z

QA tests have started for PR 2138 at commit 6058a58.

This patch merges cleanly.

andrewor14 · 2014-08-26T17:14:27Z

core/src/main/scala/org/apache/spark/SparkEnv.scala

The BlockManagerMaster actually exists on executors too. This is the consequence of bad naming, but this class is really the "agent with which the BlockManager (on both executors and the driver) communicates with the driver"

As in, you don't need to stop this only if this is the driver

As I mentioned below, only one instance of BlockManagerMaster should communicate with driver.
So far, driver itself communicates with DriverActor via SparkContext.stop invoked in user program or ApplicationMaster (Cluster mode).

SparkQA · 2014-08-26T17:36:22Z

QA tests have finished for PR 2138 at commit 6058a58.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2014-08-26T17:39:09Z

@andrewor14 Yes, Executor has a instance of BlockManagerMaster but all the important things BlockManagerMaster#stop do is send message "StopBlockManagerMaster" to DriverActor.

If we allow Executors to send the message, most of sending should be fail because DriverActor accept at most one message "StopBlockManagerMaster".

SparkQA · 2014-08-29T14:14:25Z

QA tests have started for PR 2138 at commit d5ab19a.

This patch merges cleanly.

SparkQA · 2014-08-29T15:09:20Z

QA tests have finished for PR 2138 at commit d5ab19a.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-08-29T18:31:19Z

@sarutak This assumes that the only logic in blockManagerMaster.stop() is sending the stop message to the driver. While this is true in the current code, we may add logic there in the future that also affects the executors. I think the correct approach here is to add the conditional inside BlockManagerMaster#stop itself rather than in SparkEnv.

sarutak · 2014-09-01T04:38:01Z

@andrewor14 Yeah, exactly. I'll try to implement conditional logic in BlockManagerMaster#stop itself.

…ge when sender is Driver

SparkQA · 2014-09-01T04:49:07Z

QA tests have started for PR 2138 at commit 4da8535.

This patch merges cleanly.

SparkQA · 2014-09-01T05:51:35Z

QA tests have finished for PR 2138 at commit 4da8535.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-01T08:40:29Z

core/src/main/scala/org/apache/spark/SparkEnv.scala

You don't need to past this to SparkEnv itself. Just pass isDriver to the BlockManagerMaster

SparkQA · 2014-09-01T11:49:10Z

QA tests have started for PR 2138 at commit 889e2d1.

This patch merges cleanly.

SparkQA · 2014-09-01T11:50:06Z

QA tests have finished for PR 2138 at commit 889e2d1.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf, isDriver: Boolean) extends Logging

SparkQA · 2014-09-01T11:59:13Z

QA tests have started for PR 2138 at commit 039b747.

This patch merges cleanly.

SparkQA · 2014-09-01T13:02:40Z

QA tests have finished for PR 2138 at commit 039b747.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(var driverActor: ActorRef,

andrewor14 · 2014-09-02T02:02:08Z

core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala

The format should be

class BlockManagerMaster( var driverActor... conf:... isDriver:... extends Logging {

andrewor14 · 2014-09-02T02:02:58Z

1 minor comment and this LGTM

sarutak · 2014-09-02T03:14:32Z

Thanks @andrewor14 .
I've modified format.

SparkQA · 2014-09-02T03:19:18Z

QA tests have started for PR 2138 at commit d3005fd.

This patch merges cleanly.

SparkQA · 2014-09-02T04:21:34Z

QA tests have finished for PR 2138 at commit d3005fd.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(

SparkQA · 2014-09-03T16:29:11Z

QA tests have started for PR 2138 at commit c0205b7.

This patch merges cleanly.

SparkQA · 2014-09-03T17:23:23Z

QA tests have finished for PR 2138 at commit c0205b7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(

sarutak · 2014-09-03T18:16:14Z

Jenkins, retest this please.

SparkQA · 2014-09-03T18:19:08Z

QA tests have started for PR 2138 at commit c0205b7.

This patch merges cleanly.

SparkQA · 2014-09-03T19:11:10Z

QA tests have finished for PR 2138 at commit c0205b7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(

sarutak · 2014-09-03T19:12:49Z

The failure was due to PageRank.
Jenkins, retest this please.

sarutak · 2014-09-03T19:29:14Z

Jenkins, retest this please.

SparkQA · 2014-09-03T19:34:28Z

QA tests have started for PR 2138 at commit c0205b7.

This patch merges cleanly.

SparkQA · 2014-09-03T20:31:43Z

QA tests have finished for PR 2138 at commit c0205b7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(

andrewor14 · 2014-09-03T20:43:42Z

retest this please

SparkQA · 2014-09-03T20:49:27Z

QA tests have started for PR 2138 at commit c0205b7.

This patch merges cleanly.

SparkQA · 2014-09-03T21:46:11Z

QA tests have finished for PR 2138 at commit c0205b7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockManagerMaster(

sarutak · 2014-09-03T23:40:37Z

Jenkins, retest this please.

SparkQA · 2014-09-03T23:44:14Z

QA tests have started for PR 2138 at commit c0205b7.

This patch merges cleanly.

SparkQA · 2014-09-04T00:45:19Z

QA tests have finished for PR 2138 at commit c0205b7.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
- case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)
- case class SparkListenerApplicationStart(appName: String, appId: Option[String], time: Long,
- class BlockManagerMaster(
- protected trait YarnAllocateResponse

andrewor14 · 2014-09-04T01:42:18Z

Thanks I merged this into master

…onManager etc. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#2138 from sarutak/SPARK-3233 and squashes the following commits: c0205b7 [Kousuke Saruta] Merge branch 'SPARK-3233' of github.com:sarutak/spark into SPARK-3233 064679d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3233 d3005fd [Kousuke Saruta] Modified Class definition format of BlockManagerMaster 039b747 [Kousuke Saruta] Modified style 889e2d1 [Kousuke Saruta] Modified BlockManagerMaster to be able to be past isDriver flag 4da8535 [Kousuke Saruta] Modified BlockManagerMaster#stop to send StopBlockManagerMaster message when sender is Driver 6518c3a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3233 d5ab19a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3233 6bce25c [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3233 6058a58 [Kousuke Saruta] Modified Executor not to invoke SparkEnv#stop in local mode e5ad9d3 [Kousuke Saruta] Modified Executor to stop SparnEnv at the end of itself

Modified Executor to stop SparnEnv at the end of itself

e5ad9d3

Modified Executor not to invoke SparkEnv#stop in local mode

6058a58

andrewor14 reviewed Aug 26, 2014
View reviewed changes

sarutak added 2 commits August 29, 2014 00:21

Merge branch 'master' of git://git.apache.org/spark into SPARK-3233

6bce25c

Merge branch 'master' of git://git.apache.org/spark into SPARK-3233

d5ab19a

Merge branch 'master' of git://git.apache.org/spark into SPARK-3233

6518c3a

Modified BlockManagerMaster#stop to send StopBlockManagerMaster messa…

4da8535

…ge when sender is Driver

andrewor14 reviewed Sep 1, 2014
View reviewed changes

Modified BlockManagerMaster to be able to be past isDriver flag

889e2d1

Modified style

039b747

andrewor14 reviewed Sep 2, 2014
View reviewed changes

Modified Class definition format of BlockManagerMaster

d3005fd

sarutak added 2 commits September 4, 2014 01:22

Merge branch 'master' of git://git.apache.org/spark into SPARK-3233

064679d

Merge branch 'SPARK-3233' of github.com:sarutak/spark into SPARK-3233

c0205b7

asfgit closed this in 4bba10c Sep 4, 2014

andrewor14 mentioned this pull request Sep 11, 2014

[SPARK-2572] Delete the local dir on executor automatically when using spark on Mesos. #1480

Closed

sarutak deleted the SPARK-3233 branch April 11, 2015 05:21

[SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager etc. #2138

[SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager etc. #2138

Uh oh!

Conversation

sarutak commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

andrewor14 Aug 26, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 Aug 26, 2014

Choose a reason for hiding this comment

Uh oh!

sarutak Aug 26, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

sarutak commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 29, 2014

Uh oh!

SparkQA commented Aug 29, 2014

Uh oh!

andrewor14 commented Aug 29, 2014

Uh oh!

sarutak commented Sep 1, 2014

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

andrewor14 Sep 1, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

SparkQA commented Sep 1, 2014

Uh oh!

andrewor14 Sep 2, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Sep 2, 2014

Uh oh!

sarutak commented Sep 2, 2014

Uh oh!

SparkQA commented Sep 2, 2014

Uh oh!

SparkQA commented Sep 2, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

sarutak commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

sarutak commented Sep 3, 2014

Uh oh!

sarutak commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

andrewor14 commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014