SPARK-1818 Freshen Mesos documentation #756

ash211 · 2014-05-13T06:48:46Z

Place more emphasis on using precompiled binary versions of Spark and Mesos
instead of encouraging the reader to compile from source.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source.

AmplabJenkins · 2014-05-13T06:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T06:53:07Z

Merged build started.

pwendell · 2014-05-13T07:03:46Z

docs/running-on-mesos.md

+
+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not


Incidentally I think MESOS_VERSION needs to be updated to 0.18 - mind doing that?

Updated to 0.18.1 per the pom.xml

AmplabJenkins · 2014-05-13T07:12:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T07:13:07Z

Merged build started.

AmplabJenkins · 2014-05-13T07:32:12Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T07:32:13Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14936/

pwendell · 2014-05-13T07:40:29Z

docs/running-on-mesos.md

+## Verification
+
+To verify that the Mesos cluster is ready for Spark, navigate to the Mesos master webui at port
+:5050  Confirm that all expected machines are present in the slaves tab 


Period at the end of this sentence?

pwendell · 2014-05-13T07:44:26Z

Made a few surface level comments - this is a great and much-needed update. Thanks!

AmplabJenkins · 2014-05-13T07:52:35Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T07:52:35Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14937/

At least that's how I read SparkILoop#createSparkContext()

ash211 · 2014-05-13T08:17:15Z

Made changes to cover Patrick's suggestions.

I had some trouble getting LZO libraries accessible from the Mesos executor and never actually got that working. Do you know if there's a way to add additional .jars to the Mesos executors, or do all .jars need to be bundled in the distribution tarball downloaded from e.g. HDFS?

If so I'd like to add that to the documentation too.

AmplabJenkins · 2014-05-13T08:17:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T08:18:02Z

Merged build started.

ash211 · 2014-05-13T08:44:34Z

cc: @ceteri -- we're updating the Spark documentation for running on Mesos. Given your expertise with this setup (I believe you did the video and wrote the docs below), would you mind taking a look at this documentation refresh?

This will likely end up being the documentation for the next Spark release and I'd like to get it in a great state for the big 1.0

https://www.youtube.com/watch?v=KVWMhIeKM_A
http://mesosphere.io/learn/run-spark-on-mesos

Thanks!
Andrew

AmplabJenkins · 2014-05-13T08:57:04Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T08:57:04Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14938/

maasg · 2014-05-13T11:44:39Z

docs/running-on-mesos.md

-   * `export SPARK_EXECUTOR_URI=<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>`.
-   * `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and port (default: 5050) of your Mesos master (or `zk://...` if using Mesos with ZooKeeper).
-8. To run a Spark application against the cluster, when you create your `SparkContext`, pass the string `mesos://HOST:PORT` as the master URL. In addition, you'll need to set the `spark.executor.uri` property. For example:
+# Why Mesos


I miss a chapter on "How Spark works on Mesos" that helps understanding the need of the assembly in a reachable location by the slaves and how the spark driver program becomes the 'scheduler' in Mesos terms and the tasks get delivered to the executors.

maasg · 2014-05-13T11:52:35Z

Great work.
I'd love to see some more background of the dynamics of Spark running on Mesos. It has been a tough learning experience to get our Spark + Spark Streaming running on a Mesos cluster, and I think the docs can make the life easier for 'future generations' :-)

ash211 · 2014-05-14T05:49:22Z

Hi @maasg thanks for the review!

I added a section on troubleshooting and debugging, and also a quick overview of how Spark and Mesos interaction in a How it works section.

Do those start to cover what you were looking for as an addition?

AmplabJenkins · 2014-05-14T05:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-14T05:53:05Z

Merged build started.

AmplabJenkins · 2014-05-14T05:54:23Z

Merged build finished.

AmplabJenkins · 2014-05-14T05:54:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14961/

pwendell · 2014-05-14T16:36:44Z

docs/running-on-mesos.md

+When running a shell the `spark.executor.uri` parameter is inherited from `SPARK_EXECUTOR_URI`, so
+it does not need to be redundantly passed in as a system property.
+
+{% highlight shell %}


I think this needs to be highlight bash. At least on my version of jekyll I get an error otherwise when compiling docs.

pwendell · 2014-05-14T16:45:20Z

I'm going to pull this in with the required build fix so it gets into a release candidate. This is a major improvement to the mesos docs. Please feel free to submit follow on work in a new PR.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source. Author: Andrew Ash <andrew@andrewash.com> Closes #756 from ash211/spark-1818 and squashes the following commits: 7ef3b33 [Andrew Ash] Brief explanation of the interactions between Spark and Mesos e7dea8e [Andrew Ash] Add troubleshooting and debugging section 956362d [Andrew Ash] Don't need to pass spark.executor.uri into the spark shell de3353b [Andrew Ash] Wrap to 100char 7ebf6ef [Andrew Ash] Polish on the section on Mesos Master URLs 3dcc2c1 [Andrew Ash] Use --tgz parameter of make-distribution 41b68ed [Andrew Ash] Period at end of sentence; formatting on :5050 8bf2c53 [Andrew Ash] Update site.MESOS_VERSIOn to match /pom.xml 74f2040 [Andrew Ash] SPARK-1818 Freshen Mesos documentation (cherry picked from commit d1d41cc) Signed-off-by: Patrick Wendell <pwendell@gmail.com>

ash211 · 2014-05-14T16:53:13Z

Thanks Patrick!
On May 14, 2014 9:45 AM, "Patrick Wendell" notifications@github.com wrote:

I'm going to pull this in with the required build fix so it gets into a
release candidate. This is a major improvement to the mesos docs. Please
feel free to submit follow on work in a new PR.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756#issuecomment-43105912
.

maasg · 2014-05-15T15:08:17Z

Thanks for the updates! +1 (after the facts :) )

mateiz · 2014-05-16T22:14:58Z

docs/running-on-mesos.md

+The Apache Mesos project only publishes source package releases, no binary releases.  But other
+third party projects publish binary releases that may be helpful in setting Mesos up.
+
+One of those is Mesosphere.  To install Mesos using the binary releases provided by Mesosphere:


@ash211 @pwendell, I missed this when the PR was being reviewed, but we need to list the source instructions first. We are first and foremost an Apache project, and need to tell people how to use Apache software to get their work done. Third parties can go ahead and publish binary distributions if they want, but we can't be posting that as the preferred way to install another Apache project. We do the same thing for Hadoop distros -- we say how to build against an Apache release of Hadoop, and then we have a page on building against third-party distros.

Also I would call these "third-party packages" instead of "prebuilt packages". Again, since this is going on an Apache web page, we can't be misrepresenting the Mesos project. Bits that are not built by Apache are unfortunately third-party.

Got it. I'm starting a cleanup PR that should be in within an hour.

On Fri, May 16, 2014 at 3:17 PM, Matei Zaharia notifications@github.comwrote:

In docs/running-on-mesos.md:

+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not
+require any special patches of Mesos.
+
+If you already have a Mesos cluster running, you can skip this Mesos installation step.
+
+Otherwise, installing Mesos for Spark is no different than installing Mesos for use by other
+frameworks. You can install Mesos using either prebuilt packages or by compiling from source.
+
+## Prebuilt packages
+
+The Apache Mesos project only publishes source package releases, no binary releases. But other
+third party projects publish binary releases that may be helpful in setting Mesos up.
+
+One of those is Mesosphere. To install Mesos using the binary releases provided by Mesosphere:

Also I would call these "third-party packages" instead of "prebuilt
packages". Again, since this is going on an Apache web page, we can't be
misrepresenting the Mesos project. Bits that are not built by Apache are
unfortunately third-party.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756/files#r12761962
.

#805

On Fri, May 16, 2014 at 3:41 PM, Andrew Ash andrew@andrewash.com wrote:

Got it. I'm starting a cleanup PR that should be in within an hour.

On Fri, May 16, 2014 at 3:17 PM, Matei Zaharia notifications@github.comwrote:

In docs/running-on-mesos.md:

+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not
+require any special patches of Mesos.
+
+If you already have a Mesos cluster running, you can skip this Mesos installation step.
+
+Otherwise, installing Mesos for Spark is no different than installing Mesos for use by other
+frameworks. You can install Mesos using either prebuilt packages or by compiling from source.
+
+## Prebuilt packages
+
+The Apache Mesos project only publishes source package releases, no binary releases. But other
+third party projects publish binary releases that may be helpful in setting Mesos up.
+
+One of those is Mesosphere. To install Mesos using the binary releases provided by Mesosphere:

Also I would call these "third-party packages" instead of "prebuilt
packages". Again, since this is going on an Apache web page, we can't be
misrepresenting the Mesos project. Bits that are not built by Apache are
unfortunately third-party.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756/files#r12761962
.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source. Author: Andrew Ash <andrew@andrewash.com> Closes apache#756 from ash211/spark-1818 and squashes the following commits: 7ef3b33 [Andrew Ash] Brief explanation of the interactions between Spark and Mesos e7dea8e [Andrew Ash] Add troubleshooting and debugging section 956362d [Andrew Ash] Don't need to pass spark.executor.uri into the spark shell de3353b [Andrew Ash] Wrap to 100char 7ebf6ef [Andrew Ash] Polish on the section on Mesos Master URLs 3dcc2c1 [Andrew Ash] Use --tgz parameter of make-distribution 41b68ed [Andrew Ash] Period at end of sentence; formatting on :5050 8bf2c53 [Andrew Ash] Update site.MESOS_VERSIOn to match /pom.xml 74f2040 [Andrew Ash] SPARK-1818 Freshen Mesos documentation

SPARK-1818 Freshen Mesos documentation

74f2040

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source.

pwendell reviewed May 13, 2014
View reviewed changes

Update site.MESOS_VERSIOn to match /pom.xml

8bf2c53

pwendell reviewed May 13, 2014
View reviewed changes

ash211 added 2 commits May 13, 2014 00:47

Period at end of sentence; formatting on :5050

41b68ed

Use --tgz parameter of make-distribution

3dcc2c1

ash211 added 3 commits May 13, 2014 01:06

Polish on the section on Mesos Master URLs

7ebf6ef

Wrap to 100char

de3353b

Don't need to pass spark.executor.uri into the spark shell

956362d

At least that's how I read SparkILoop#createSparkContext()

maasg reviewed May 13, 2014
View reviewed changes

Add troubleshooting and debugging section

e7dea8e

Brief explanation of the interactions between Spark and Mesos

7ef3b33

pwendell reviewed May 14, 2014
View reviewed changes

asfgit closed this in d1d41cc May 14, 2014

mateiz reviewed May 16, 2014
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPARK-1818 Freshen Mesos documentation #756

SPARK-1818 Freshen Mesos documentation #756

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

pwendell May 13, 2014

ash211 May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

pwendell May 13, 2014

pwendell commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

maasg May 13, 2014

maasg commented May 13, 2014

ash211 commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

pwendell May 14, 2014

pwendell commented May 14, 2014

ash211 commented May 14, 2014

maasg commented May 15, 2014

mateiz May 16, 2014

mateiz May 16, 2014

ash211 May 16, 2014

ash211 May 16, 2014


		# Installing Mesos

		Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not

SPARK-1818 Freshen Mesos documentation #756

SPARK-1818 Freshen Mesos documentation #756

Conversation

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

Choose a reason for hiding this comment

pwendell commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

ash211 commented May 13, 2014

AmplabJenkins commented May 13, 2014

AmplabJenkins commented May 13, 2014

Choose a reason for hiding this comment

maasg commented May 13, 2014

ash211 commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

Choose a reason for hiding this comment

pwendell commented May 14, 2014

ash211 commented May 14, 2014

maasg commented May 15, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment