[SPARK-4048] Enhance and extend hadoop-provided profile. #2982

vanzin · 2014-10-28T19:50:26Z

This change does a few things to make the hadoop-provided profile more useful:

Create new profiles for other libraries / services that might be provided by the infrastructure
Simplify and fix the poms so that the profiles are only activated while building assemblies.
Fix tests so that they're able to run when the profiles are activated
Add a new env variable to be used by distributions that use these profiles to provide the runtime
classpath for Spark jobs and daemons.

vanzin · 2014-10-28T19:51:52Z

Notes:

I ran the unit tests and also deployed the assemblies built with all profiles enabled internally, and made sure everything still worked.
The individual commits have more explanations for each part of the overall patch, so take a look at them when in doubt about something.

SparkQA · 2014-10-28T19:55:19Z

Test build #22371 has started for PR 2982 at commit d261346.

This patch merges cleanly.

SparkQA · 2014-10-28T21:27:10Z

Test build #22371 has finished for PR 2982 at commit d261346.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class VectorTransformer(object):
- class Normalizer(VectorTransformer):
- class JavaModelWrapper(VectorTransformer):
- class StandardScalerModel(JavaModelWrapper):
- class StandardScaler(object):
- class HashingTF(object):
- class IDFModel(JavaModelWrapper):
- class IDF(object):
- class Word2VecModel(JavaModelWrapper):

AmplabJenkins · 2014-10-28T21:27:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22371/
Test PASSed.

qiaohaijun · 2014-11-01T14:13:21Z

+1

SparkQA · 2014-11-04T00:44:55Z

Test build #22836 has started for PR 2982 at commit a89b833.

This patch merges cleanly.

SparkQA · 2014-11-04T00:45:52Z

Test build #22836 has finished for PR 2982 at commit a89b833.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-04T00:45:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22836/
Test FAILed.

SparkQA · 2014-11-04T00:55:35Z

Test build #22838 has started for PR 2982 at commit 93b4789.

This patch merges cleanly.

SparkQA · 2014-11-04T02:41:17Z

Test build #22838 has finished for PR 2982 at commit 93b4789.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class NullType(PrimitiveType):
- case class ScalaUdfBuilder[T: TypeTag](f: AnyRef)

AmplabJenkins · 2014-11-04T02:41:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22838/
Test PASSed.

SparkQA · 2014-11-04T18:05:17Z

Test build #22884 has started for PR 2982 at commit 615118f.

This patch merges cleanly.

SparkQA · 2014-11-04T19:52:47Z

Test build #22884 has finished for PR 2982 at commit 615118f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RDDFunctions[T: ClassTag](self: RDD[T]) extends Serializable

AmplabJenkins · 2014-11-04T19:52:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22884/
Test PASSed.

SparkQA · 2014-11-04T20:22:36Z

Test build #22893 has started for PR 2982 at commit 414e1f0.

This patch merges cleanly.

SparkQA · 2014-11-04T22:12:32Z

Test build #22893 has finished for PR 2982 at commit 414e1f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-04T22:12:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22893/
Test PASSed.

SparkQA · 2014-11-05T22:32:36Z

Test build #22952 has started for PR 2982 at commit a0f538c.

This patch merges cleanly.

SparkQA · 2014-11-06T00:20:50Z

Test build #22952 has finished for PR 2982 at commit a0f538c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T00:20:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22952/
Test PASSed.

SparkQA · 2014-11-06T20:34:54Z

Test build #23016 has started for PR 2982 at commit f298adc.

This patch merges cleanly.

SparkQA · 2014-11-06T22:21:09Z

Test build #23016 has finished for PR 2982 at commit f298adc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T22:21:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23016/
Test PASSed.

tdas · 2014-11-07T21:00:06Z

streaming/pom.xml

@@ -41,10 +41,6 @@
      <version>${project.version}</version>
    </dependency>
    <dependency>
-      <groupId>org.eclipse.jetty</groupId>


Why was this removed?

I did not see any direct dependency on jetty from the streaming code. I'll double check, though.

So, streaming does not depend on Jetty, but it does use servlet APIs. Since Spark doesn't pull in any servlet API dependency directly, and instead uses the transitive one provided by Jetty, which is some Jetty repackaging of servlet APIs instead of some common dependency. (BTW it works without the explicit dependency because now it was coming as a transitive dependency of spark-core.)

In light of that I'll restore this dependency, even though I don't really like it very much (it's pulling a dependency that is too coarse for what it needs).

SparkQA · 2014-11-12T03:25:13Z

Test build #23243 has started for PR 2982 at commit 3ee72b3.

This patch merges cleanly.

SparkQA · 2014-11-12T03:39:56Z

Test build #23244 has started for PR 2982 at commit f1173d6.

This patch merges cleanly.

vanzin · 2014-11-12T04:14:34Z

Friendly ping. I still have to double check TD's question, but even pending that, the code builds and runs correctly.

SparkQA · 2014-11-12T05:16:04Z

Test build #23243 has finished for PR 2982 at commit 3ee72b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-12T05:16:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23243/
Test PASSed.

pwendell · 2015-01-08T06:48:52Z

core/src/main/scala/org/apache/spark/util/Utils.scala

+    // compute-classpath.{cmd,sh} and makes all needed jars available to child processes
+    // when the assembly is built with the "*-provided" profiles enabled.
+    if (sys.props.contains("spark.testing")) {
+      environment.put("SPARK_DIST_CLASSPATH", sys.props("java.class.path"))


Is this needed in all cases or only when you are actually running tests with a "hadoop provided" build and you've supplied you own SPARK_DIST_CLASSPATH for the test JVM. If the latter is true, can we just propagate the value of SPARK_DIST_CLASSPATH to the child from the parent? It's a bit weird here because you're setting SPARK_DIST_CLASSPATH to also include all of the normal spark classes.

Then you would just check if SPARK_DIST_CLASSPATH is set and if so, you'd propagate its value to the child environment.

I used to run into problems with tests spawning child processes that referenced hadoop classes, and those not being available when the "hadoop-provided" profile was enabled. Let me try without this and see how it goes, since I noticed some changes in the configuration of the test plugins in the root pom.

Even if it is still a problem, my question was, can we just set SPARK_DIST_CLASSPATH in the child to the value of SPARK_DIST_CLASSPATH in the parent? This is instead of setting it to the value of java.class.path.

But SPARK_DIST_CLASSPATH is not set in the parent (the parent being the surefire/scalatest runner, in this case). The goal of this code is to add the classpath built by those (which includes all the dependencies from the pom) to the child processes without having to modify the test code to manually do it.

pwendell · 2015-01-08T06:52:16Z

Hey @vanzin. I left a comment about the test set-up but overall this looks good. Fine to keep SPARK_DIST_CLASSPATH as is. It seems reasonable to have a way for distributions to customize the default global classpath behavior. My main hesitation was that we used to have SPARK_CLASSPATH and many users would just add their jars, etc using that variable and it didn't work well in mutli-tenant environments. Now we have better mechanisms to point users to, so this seems fine.

Conflicts: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

SparkQA · 2015-01-08T20:32:41Z

Test build #25258 has started for PR 2982 at commit 4e38f4e.

This patch merges cleanly.

SparkQA · 2015-01-08T20:39:00Z

Test build #25258 has finished for PR 2982 at commit 4e38f4e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-08T20:39:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25258/
Test FAILed.

SparkQA · 2015-01-08T20:52:44Z

Test build #25263 has started for PR 2982 at commit eb228c0.

This patch merges cleanly.

SparkQA · 2015-01-08T22:31:58Z

Test build #25263 has finished for PR 2982 at commit eb228c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-08T22:32:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25263/
Test PASSed.

pwendell · 2015-01-08T23:17:39Z

yarn/pom.xml

-        <groupId>org.apache.maven.plugins</groupId>
-        <artifactId>maven-install-plugin</artifactId>
-        <configuration>
-          <skip>true</skip>


Is this change a separate issue?

vanzin@1adf91c

SparkQA · 2015-01-08T23:27:37Z

Test build #25273 has started for PR 2982 at commit 82eb688.

This patch merges cleanly.

SparkQA · 2015-01-09T01:07:43Z

Test build #25273 has finished for PR 2982 at commit 82eb688.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-09T01:07:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25273/
Test PASSed.

pwendell · 2015-01-09T01:14:57Z

Thanks @vanzin for this cleanup - I'll merge this.

After #2982 (SPARK-4048) we rely on the newer HBase packaging format.

vanzin force-pushed the SPARK-4048 branch from 230dd30 to a89b833 Compare November 4, 2014 00:37

tdas reviewed Nov 7, 2014
View reviewed changes

pwendell reviewed Jan 8, 2015
View reviewed changes

Marcelo Vanzin added 2 commits January 8, 2015 12:29

Alternative way to propagate test classpath to child processes.

9ef79a3

Merge branch 'master' into SPARK-4048

4e38f4e

Conflicts: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

Fix borked merge.

eb228c0

pwendell reviewed Jan 8, 2015
View reviewed changes

Add a comment.

82eb688

asfgit closed this in 48cecf6 Jan 9, 2015

vanzin deleted the SPARK-4048 branch January 9, 2015 18:12

This was referenced Jan 16, 2015

[HOTFIX]: Minor clean up regarding skipped artifacts in build files. #4078

Closed

[SPARK-5411] Allow SparkListeners to be specified in SparkConf and loaded when creating SparkContext #4111

Closed

asfgit pushed a commit that referenced this pull request Mar 3, 2015

HOTFIX: Bump HBase version in MapR profiles.

1aa8461

After #2982 (SPARK-4048) we rely on the newer HBase packaging format.

asfgit pushed a commit that referenced this pull request Mar 3, 2015

HOTFIX: Bump HBase version in MapR profiles.

165ff36

After #2982 (SPARK-4048) we rely on the newer HBase packaging format.

[SPARK-4048] Enhance and extend hadoop-provided profile. #2982

[SPARK-4048] Enhance and extend hadoop-provided profile. #2982

Conversation

vanzin commented Oct 28, 2014

vanzin commented Oct 28, 2014

SparkQA commented Oct 28, 2014

SparkQA commented Oct 28, 2014

AmplabJenkins commented Oct 28, 2014

qiaohaijun commented Nov 1, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 5, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

SparkQA commented Nov 6, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 12, 2014

SparkQA commented Nov 12, 2014

vanzin commented Nov 12, 2014

SparkQA commented Nov 12, 2014

AmplabJenkins commented Nov 12, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

AmplabJenkins commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

AmplabJenkins commented Jan 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 8, 2015

SparkQA commented Jan 9, 2015

AmplabJenkins commented Jan 9, 2015

pwendell commented Jan 9, 2015