[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

tnachen · 2015-12-02T01:37:38Z

Adding more documentation about submitting jobs with mesos cluster mode.

tnachen · 2015-12-02T01:41:00Z

andrewor14 · 2015-12-02T01:45:04Z

docs/submitting-applications.md

@@ -132,9 +144,10 @@ The master URL passed to Spark can be in one of the following formats:
 <tr><td> <code>mesos://HOST:PORT</code> </td><td> Connect to the given <a href="running-on-mesos.html">Mesos</a> cluster.
        The port must be whichever one your is configured to use, which is 5050 by default.
        Or, for a Mesos cluster using ZooKeeper, use <code>mesos://zk://...</code>.
+        To submit with <code>--deploy-mode cluster</code>, the HOST:PORT should be configured to connect to the MesosClusterDispatcher.


do they know at this point what a MesosClusterDispatcher is?

@tnachen can you update this assuming the user doesn't know anything about the dispatcher?

andrewor14 · 2015-12-02T01:45:57Z

looks alright

SparkQA · 2015-12-02T02:07:38Z

Test build #47028 has finished for PR 10086 at commit 448d706.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-12-03T18:44:08Z

@dragos you mentioned you had difficulty running mesos in cluster mode. Does this additional documentation help? What were the issues you ran into?

tnachen · 2015-12-03T19:29:02Z

@andrewor14 I don't tihnk this doc helps his problem, he's running into issues where Spark configuration is not being passed down. I'm pushing a new update here.

SparkQA · 2015-12-03T20:01:35Z

Test build #47158 has finished for PR 10086 at commit 5d612d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2015-12-04T05:27:24Z

@andrewor14 pushed an update today, PTAL

dragos · 2015-12-04T14:10:39Z

Thanks, this is a good example! Can you please update also running-on-mesos.md, that part is where most people running on Mesos will look first. I think an example like this one, and one showing how to start the dispatcher would be great.

dragos · 2015-12-04T14:12:39Z

@andrewor14 the issue in the other PR is not due to missing docs, unfortunately. I will post my errors in that PR (I reached out to Tim via Slack earlier, but maybe it makes sense to have such discussions in the open)

tnachen · 2015-12-08T07:26:16Z

@dragos just updated the docs

SparkQA · 2015-12-08T08:02:27Z

Test build #47318 has finished for PR 10086 at commit 952f116.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * If you like to run the MesosClusterDispatcher with Marathon, you need to run the MesosClusterDispatcher in the foreground (i.e:bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher).\n

dragos · 2015-12-08T08:52:32Z

LGTM

srowen · 2015-12-08T12:18:24Z

docs/running-on-mesos.md

-passing in the Mesos master url (e.g: mesos://host:5050).
+passing in the Mesos master url (e.g: mesos://host:5050). This starts the MesosClusterDispatcher as an daemon running on the host.
+
+If you like to run the MesosClusterDispatcher with Marathon, you need to run the MesosClusterDispatcher in the foreground (i.e: `bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher`).


Nits: "an daemon" -> "a daemon", url -> URL, "i.e" -> "i.e.". You might back-tick MesosClusterDispatcher too.

yeah, I would backtick the dispatcher

andrewor14 · 2015-12-11T19:03:53Z

@tnachen can you address the comments? I'll merge this once you do so.

srowen · 2015-12-15T11:41:18Z

Ping @tnachen

tnachen · 2015-12-15T13:58:50Z

@srowen @andrewor14 sorry for the delay, it's updated now.

SparkQA · 2015-12-15T14:25:22Z

Test build #47741 has finished for PR 10086 at commit 7f8d7b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * If you like to run theMesosClusterDispatcherwith Marathon, you need to run theMesosClusterDispatcherin the foreground (i.e:bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher).\n

srowen · 2015-12-15T20:45:34Z

Last but not least, should we go ahead and make a JIRA for this? it's pretty minor but not trivial

tnachen · 2015-12-16T02:09:02Z

Created jira and updated title @srowen

andrewor14 · 2015-12-16T02:19:46Z

Merged into master and 1.6

…sos cluster mode. Adding more documentation about submitting jobs with mesos cluster mode. Author: Timothy Chen <tnachen@gmail.com> Closes #10086 from tnachen/mesos_supervise_docs. (cherry picked from commit c2de99a) Signed-off-by: Andrew Or <andrew@databricks.com>

tnachen changed the title ~~Add documentation about submitting Spark with mesos cluster mode.~~ [MESOS]Add documentation about submitting Spark with mesos cluster mode. Dec 2, 2015

andrewor14 reviewed Dec 2, 2015
View reviewed changes

tnachen force-pushed the mesos_supervise_docs branch from 448d706 to 5d612d0 Compare December 3, 2015 19:29

tnachen force-pushed the mesos_supervise_docs branch from 5d612d0 to 952f116 Compare December 8, 2015 07:22

srowen reviewed Dec 8, 2015
View reviewed changes

tnachen added 2 commits December 15, 2015 21:58

Add documentation about submitting Spark with mesos cluster mode.

188a5c3

Update running on mesos doc as well.

7f8d7b6

tnachen force-pushed the mesos_supervise_docs branch from 952f116 to 7f8d7b6 Compare December 15, 2015 13:58

tnachen changed the title ~~[MESOS]Add documentation about submitting Spark with mesos cluster mode.~~ [SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. Dec 16, 2015

asfgit closed this in c2de99a Dec 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

tnachen commented Dec 2, 2015

tnachen commented Dec 2, 2015

andrewor14 Dec 2, 2015

andrewor14 Dec 3, 2015

andrewor14 commented Dec 2, 2015

SparkQA commented Dec 2, 2015

andrewor14 commented Dec 3, 2015

tnachen commented Dec 3, 2015

SparkQA commented Dec 3, 2015

tnachen commented Dec 4, 2015

dragos commented Dec 4, 2015

dragos commented Dec 4, 2015

tnachen commented Dec 8, 2015

SparkQA commented Dec 8, 2015

dragos commented Dec 8, 2015

srowen Dec 8, 2015

andrewor14 Dec 11, 2015

andrewor14 commented Dec 11, 2015

srowen commented Dec 15, 2015

tnachen commented Dec 15, 2015

SparkQA commented Dec 15, 2015

srowen commented Dec 15, 2015

tnachen commented Dec 16, 2015

andrewor14 commented Dec 16, 2015

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

Conversation

tnachen commented Dec 2, 2015

tnachen commented Dec 2, 2015

andrewor14 Dec 2, 2015

Choose a reason for hiding this comment

andrewor14 Dec 3, 2015

Choose a reason for hiding this comment

andrewor14 commented Dec 2, 2015

SparkQA commented Dec 2, 2015

andrewor14 commented Dec 3, 2015

tnachen commented Dec 3, 2015

SparkQA commented Dec 3, 2015

tnachen commented Dec 4, 2015

dragos commented Dec 4, 2015

dragos commented Dec 4, 2015

tnachen commented Dec 8, 2015

SparkQA commented Dec 8, 2015

dragos commented Dec 8, 2015

srowen Dec 8, 2015

Choose a reason for hiding this comment

andrewor14 Dec 11, 2015

Choose a reason for hiding this comment

andrewor14 commented Dec 11, 2015

srowen commented Dec 15, 2015

tnachen commented Dec 15, 2015

SparkQA commented Dec 15, 2015

srowen commented Dec 15, 2015

tnachen commented Dec 16, 2015

andrewor14 commented Dec 16, 2015