[SPARK-11714][Mesos] Make Spark on Mesos honor port restrictions on coarse grain mode #11157

skonto · 2016-02-10T20:56:12Z

Make mesos coarse grained scheduler accept port offers and pre-assign ports

Previous attempt was for fine grained: #10808

andrewor14 · 2016-02-10T21:35:22Z

add to whitelist

andrewor14 · 2016-02-10T21:35:41Z

@skonto can you add a link in the description to your other PR?

skonto · 2016-02-11T09:21:20Z

Done @andrewor14. I will resolve conflicts.

skonto · 2016-02-11T12:48:17Z

Jenkins test this please

Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackendSuite.scala

SparkQA · 2016-02-11T14:45:21Z

Test build #51096 has started for PR 11157 at commit a4e575d.

SparkQA · 2016-02-11T15:33:58Z

Test build #51095 has finished for PR 11157 at commit 53f2ced.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

shaneknapp · 2016-02-11T16:08:19Z

jenkins, test this please

SparkQA · 2016-02-11T18:16:43Z

Test build #51108 has finished for PR 11157 at commit a4e575d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Grouping(child: Expression) extends Expression with Unevaluable
- case class GroupingID(groupByExprs: Seq[Expression]) extends Expression with Unevaluable
- class ContinuousQueryManager(sqlContext: SQLContext)
- class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
- class FileStreamSource(
- trait HadoopFsRelationProvider extends StreamSourceProvider
- abstract class ContinuousQueryListener

skonto · 2016-02-11T18:48:19Z

Jenkins, test this please

SparkQA · 2016-02-11T20:53:27Z

Test build #51123 has finished for PR 11157 at commit a4e575d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Grouping(child: Expression) extends Expression with Unevaluable
- case class GroupingID(groupByExprs: Seq[Expression]) extends Expression with Unevaluable
- class ContinuousQueryManager(sqlContext: SQLContext)
- class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
- class FileStreamSource(
- trait HadoopFsRelationProvider extends StreamSourceProvider
- abstract class ContinuousQueryListener

skonto · 2016-02-11T22:37:11Z

Need to fix the test. Multiple executors per slave need different handling.

skonto · 2016-02-29T10:29:37Z

Still working on this had to get aligned with @mgummelt.

tnachen · 2016-03-04T23:28:46Z

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala

            partitionResources(afterCPUResources.asJava, "mem", taskMemory)
+          val (resourcesLeft, portResourcesToUse) = remainingMemResources
+            .partition {r => ! ( r.getType == Value.Type.RANGES &  r.getName == "ports" )}


I think there is still style problems here.

yes havent finished it.... port resources need to be managed as cpus eg. keep global logistics... but what exactly is the problem for you?

tnachen · 2016-03-04T23:30:10Z

Can you fix the whitespacing in general in your patch? There are quite some extra whitespaces in the Utils.scala

skonto · 2016-03-07T17:18:20Z

ok i will fix it thnx... WIP

skonto · 2016-03-12T01:58:24Z

updated the PR @tnachen @mgummelt, @dragos ready for review....
This follows our design discussions:
Design choices:

At offer evaluation step check (scheduler side) if we have enough ports to start an executor in an offer:

if yes, we start it and accept the offer
otherwise decline the offer.
If a user asks for a specific port like in the case of spark.executor.port, then we cannot make available the same port to all executors of the same job, due to mesos limitations (all slave ports are shared).

Our proposed solution is to start one executor if the user sets a value to spark.executor.port since there is only one port available with that value.

Spark Cluster framework port honoring will be fixed in a new PR, to make things easier and shorter for review.
One concern i have and im testing is compatibility between frameworks running in coarse mode and fain grain mode on the same cluster.

SparkQA · 2016-03-12T01:59:10Z

Test build #52982 has finished for PR 11157 at commit d1cb53b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-12T02:04:08Z

Test build #52983 has finished for PR 11157 at commit 33d74aa.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-12T02:09:00Z

Test build #52984 has finished for PR 11157 at commit 731c2f0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-12T14:51:42Z

Test build #53002 has finished for PR 11157 at commit 023eedc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-12T19:10:26Z

Test build #53007 has finished for PR 11157 at commit c6ceb8b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2016-08-10T18:14:28Z

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala

@@ -212,12 +212,32 @@ class MesosCoarseGrainedSchedulerBackendSuite extends SparkFunSuite
      .registerDriverWithShuffleService(anyString, anyInt, anyLong, anyLong)
  }

-  test("mesos kills an executor when told") {
+  test("Port offer decline when there is no appropriate range") {


Can we get a test where the offer contains the requested ports? We should both that the offer is accepted, and that the launched task contains the requested port as a resource.

skonto · 2016-08-10T22:09:43Z

@mgummelt added the extra test needed and fixed the rest.

SparkQA · 2016-08-11T00:19:52Z

Test build #63559 has finished for PR 11157 at commit b46b9d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2016-08-11T00:24:02Z

@mgummelt : ready for merge what do you think?

mgummelt · 2016-08-11T17:44:46Z

LGTM.

@srowen Can we get this merged into master?

srowen · 2016-08-11T22:43:11Z

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala

+    if (requestedPorts.isEmpty) {
+      (offeredResources, List[Resource]())
+    }
+    else {


Can we do a quick minor style pass, since that's all I can comment on really -- pull this else up to the previous line for consistency.

skonto · 2016-08-14T18:51:24Z

@srowen ready

SparkQA · 2016-08-14T20:53:39Z

Test build #63755 has finished for PR 11157 at commit 3bc31cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2016-08-15T08:38:53Z

@srowen could you merge this pls?

srowen · 2016-08-15T08:55:44Z

Merged to master

mgummelt · 2016-08-15T18:50:37Z

woot

sun-rui · 2016-08-16T02:18:52Z

has this change be documented to spark on mesos guide?

skonto · 2016-08-16T13:33:24Z

@sun-rui Will do in a separate pr shortly. There was a question for that a few comments above. Thanx for mention it again.

skonto · 2016-08-16T16:40:08Z

@sun-rui here you are: #14667
@mgummelt check if it is clear enough pls.

honour ports

53f2ced

tnachen reviewed Mar 4, 2016
View reviewed changes

skonto force-pushed the honour_ports_coarse branch 2 times, most recently from 33d74aa to 731c2f0 Compare March 12, 2016 02:02

skonto force-pushed the honour_ports_coarse branch from 731c2f0 to 023eedc Compare March 12, 2016 12:32

manage port resources correctly

c6ceb8b

skonto force-pushed the honour_ports_coarse branch from 023eedc to c6ceb8b Compare March 12, 2016 17:07

mgummelt reviewed Aug 10, 2016
View reviewed changes

fix tests

b46b9d4

skonto force-pushed the honour_ports_coarse branch from edf9320 to b46b9d4 Compare August 10, 2016 22:07

srowen reviewed Aug 11, 2016
View reviewed changes

fix

3bc31cf

asfgit closed this in 1a028bd Aug 15, 2016

[SPARK-11714][Mesos] Make Spark on Mesos honor port restrictions on coarse grain mode #11157

[SPARK-11714][Mesos] Make Spark on Mesos honor port restrictions on coarse grain mode #11157

Conversation

skonto commented Feb 10, 2016 • edited

andrewor14 commented Feb 10, 2016

andrewor14 commented Feb 10, 2016

skonto commented Feb 11, 2016

skonto commented Feb 11, 2016

SparkQA commented Feb 11, 2016

SparkQA commented Feb 11, 2016

shaneknapp commented Feb 11, 2016

SparkQA commented Feb 11, 2016

skonto commented Feb 11, 2016

SparkQA commented Feb 11, 2016

skonto commented Feb 11, 2016

skonto commented Feb 29, 2016

tnachen Mar 4, 2016

Choose a reason for hiding this comment

skonto Mar 7, 2016

Choose a reason for hiding this comment

tnachen commented Mar 4, 2016

skonto commented Mar 7, 2016

skonto commented Mar 12, 2016

SparkQA commented Mar 12, 2016

SparkQA commented Mar 12, 2016

SparkQA commented Mar 12, 2016

SparkQA commented Mar 12, 2016

SparkQA commented Mar 12, 2016

mgummelt Aug 10, 2016 • edited

Choose a reason for hiding this comment

skonto Aug 10, 2016

Choose a reason for hiding this comment

skonto commented Aug 10, 2016

SparkQA commented Aug 11, 2016

skonto commented Aug 11, 2016

mgummelt commented Aug 11, 2016

srowen Aug 11, 2016

Choose a reason for hiding this comment

skonto Aug 12, 2016

Choose a reason for hiding this comment

skonto commented Aug 14, 2016

SparkQA commented Aug 14, 2016

skonto commented Aug 15, 2016

srowen commented Aug 15, 2016

mgummelt commented Aug 15, 2016

sun-rui commented Aug 16, 2016

skonto commented Aug 16, 2016 • edited

skonto commented Aug 16, 2016

skonto commented Feb 10, 2016 •

edited

mgummelt Aug 10, 2016 •

edited

skonto commented Aug 16, 2016 •

edited