Skip to content
Permalink
Browse files

[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provide…

…d' dependency, not package it

## What changes were proposed in this pull request?

Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency.

Packaging Spark makes the app much bigger by tens of megabytes. It can also bring in conflicting dependencies that wouldn't otherwise be a problem. https://issues.apache.org/jira/browse/SPARK-26146 was what reminded me of this.

## How was this patch tested?

Doc build

Closes #23938 from srowen/Provided.

Authored-by: Sean Owen <sean.owen@databricks.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(cherry picked from commit 3909223)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
  • Loading branch information...
srowen committed Mar 5, 2019
1 parent 3ece965 commit c32662877d90b379df87cb356c5d32b0bd0f4943
Showing with 4 additions and 1 deletion.
  1. +1 −0 docs/cloud-integration.md
  2. +1 −0 docs/quick-start.md
  3. +2 −1 docs/streaming-programming-guide.md
@@ -87,6 +87,7 @@ is set to the chosen version of Spark:
<groupId>org.apache.spark</groupId>
<artifactId>hadoop-cloud_2.11</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
...
</dependencyManagement>
@@ -336,6 +336,7 @@ Note that Spark artifacts are tagged with a Scala version.
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
@@ -385,11 +385,12 @@ Similar to Spark, Spark Streaming is available through Maven Central. To write y
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
<scope>provided</scope>
</dependency>
</div>
<div data-lang="SBT" markdown="1">

libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}"
libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}" % "provided"
</div>
</div>

0 comments on commit c326628

Please sign in to comment.
You can’t perform that action at this time.