[SPARK-13579][build][test-maven] Stop building the main Spark assembly. #11796

vanzin · 2016-03-17T21:36:00Z

This change modifies the "assembly/" module to just copy needed
dependencies to its build directory, and modifies the packaging
script to pick those up (and remove duplicate jars packages in the
examples module).

I also made some minor adjustments to dependencies to remove some
test jars from the final packaging, and remove jars that conflict with each
other when packaged separately (e.g. servlet api).

Also note that this change restores guava in applications' classpaths, even
though it's still shaded inside Spark. This is now needed for the Hadoop
libraries that are packaged with Spark, which now are not processed by
the shade plugin.

vanzin · 2016-03-17T21:36:34Z

This is probably not ready yet, although it works for me locally. I want to run the patch through jenkins (sbt and maven) to see how well it works out.

SparkQA · 2016-03-17T21:41:07Z

Test build #53471 has finished for PR 11796 at commit f679e14.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-03-17T22:31:22Z

sql/hive/pom.xml

@@ -238,36 +238,13 @@
             <configuration>
               <sources>
                 <source>v${hive.version.short}/src/main/scala</source>
-                 <source>${project.build.directory/generated-sources/antlr</source>
+                 <source>${project.build.directory}/generated-sources/antlr</source>


Nice catch :)

It didn't seem to be affecting the build, though. I'm actually wondering if this line is needed - the parser seems to be part of the catalyst module, not this one.

Yeah you are right about that. So this could actually be removed. Lemme know, I can also submit a PR for this.

I'll remove it, but since this patch may take some more iterations before it's ready, feel free to do it as part of another patch too.

SparkQA · 2016-03-17T23:59:35Z

Test build #53473 has finished for PR 11796 at commit 5d1e259.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-18T00:01:21Z

Test build #53472 has finished for PR 11796 at commit 54336b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-18T20:45:18Z

Test build #53549 has finished for PR 11796 at commit 5ec6722.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-19T01:02:52Z

Test build #53576 has finished for PR 11796 at commit a83afeb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-21T16:34:21Z

Hmm, log files are gone and tests pass locally... sigh. retest this please

SparkQA · 2016-03-21T18:39:00Z

Test build #53683 has finished for PR 11796 at commit a83afeb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-22T05:06:21Z

Test build #53735 has finished for PR 11796 at commit 5d517f0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-22T17:29:40Z

These tests fail sporadically; I think it's some weird sbt dependency resolution issue that's causing different hadoop versions to get mixed up. e.g.

16/03/21 19:43:11.830 redirect stderr for command ./bin/spark-submit INFO Utils: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
16/03/21 19:43:11.830 redirect stderr for command ./bin/spark-submit INFO Utils:    at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54)
16/03/21 19:43:11.830 redirect stderr for command ./bin/spark-submit INFO Utils:    at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42)
16/03/21 19:43:11.830 redirect stderr for command ./bin/spark-submit INFO Utils:    at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:118)

That's not the whole stack trace, but there's only hadoop stuff in there, so it's not Spark calling some method that was removed.

vanzin · 2016-03-22T18:04:06Z

The offending method (void addDeprecations(DeprecationDelta[])) seems to have been added in Hadoop 2.3 and seems to exist in all versions since. Somehow we're getting hadoop 2.2 jars in the classpath?

SparkQA · 2016-03-22T20:58:14Z

Test build #53800 has finished for PR 11796 at commit 83fa3a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-22T21:12:16Z

Weird jline error, trying to see if I can figure it out... but the tests I want to fail passed. retest this please.

vanzin · 2016-03-22T21:44:14Z

jline error looks weirdly similar. Old version of jline (0.9) has class jline.Terminal, while newer versions have an interface.

2016-03-22 13:00:57.304 - stderr> java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
2016-03-22 13:00:57.306 - stderr>   at jline.TerminalFactory.create(TerminalFactory.java:101)
2016-03-22 13:00:57.306 - stderr>   at jline.TerminalFactory.get(TerminalFactory.java:158)
2016-03-22 13:00:57.306 - stderr>   at jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
2016-03-22 13:00:57.306 - stderr>   at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)

SparkQA · 2016-03-23T00:33:38Z

Test build #53818 has finished for PR 11796 at commit 83fa3a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-23T02:08:35Z

I'm a little at a loss at what's wrong. Copy & pasting the commands from the logs and running them on the same jenkins machine works (I don't get that exception). Running javap for the Configuration class with the command's classpath shows the new class, with the method that the logs complain about...

SparkQA · 2016-03-23T02:34:06Z

Test build #53874 has finished for PR 11796 at commit b5b8dd1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T04:41:28Z

Test build #53871 has finished for PR 11796 at commit 0339263.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T04:50:14Z

Test build #53875 has finished for PR 11796 at commit 6c4ed0a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T17:15:52Z

Test build #53948 has finished for PR 11796 at commit e37886b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T19:48:07Z

Test build #53949 has finished for PR 11796 at commit 55aec21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-23T19:52:22Z

yay, but I'm still worried about the previous failures. will test on maven now.

vanzin · 2016-03-23T19:53:02Z

retest this please

vanzin · 2016-03-23T22:39:28Z

the maven build seems stuck... :-/ I'll revert to sbt and add some patches since I want to try to hit the previous failures.

JoshRosen · 2016-03-31T17:57:32Z

core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala

@@ -201,24 +201,29 @@ class FileAppenderSuite extends SparkFunSuite with BeforeAndAfter with Logging {

    // Make sure only logging errors


What's the deal with these logging changes? Got a short summary?

For some reason these tests failed in one of my runs. And when they failed, they used to leave the error level at "ERROR", which basically means all tests that ran after these didn't write any logs.

JoshRosen · 2016-03-31T18:18:24Z

Took a quick pass and have a few clarifying questions. This looks like it's in good shape overall, though.

SparkQA · 2016-03-31T19:04:44Z

Test build #54652 has finished for PR 11796 at commit 9275ea6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-31T19:22:29Z

@JoshRosen if you want to take a look at the above failure, it seems related to running the build with jdk8. retest this please

JoshRosen · 2016-03-31T19:24:35Z

Hmm, that's unfortunate. Let me flip the switch to move the PR builder back to Java 7...

JoshRosen · 2016-03-31T19:30:36Z

Jenkins, retest this please.

SparkQA · 2016-03-31T21:44:58Z

Test build #54656 has finished for PR 11796 at commit 9275ea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-31T21:56:01Z

Test build #54657 has finished for PR 11796 at commit 9275ea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-04-04T17:15:27Z

Hi @JoshRosen , any more comments?

SparkQA · 2016-04-04T20:03:31Z

Test build #54865 has finished for PR 11796 at commit b5d4292.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-04-04T20:35:19Z

bin/spark-class

 if [ -f "${SPARK_HOME}/RELEASE" ]; then
-  SPARK_JARS_DIR="${SPARK_HOME}/lib"
+  SPARK_JARS_DIR="${SPARK_HOME}/jars"
 else
  SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION"


I think you also have to append /jars here?

JoshRosen · 2016-04-04T23:30:22Z

LGTM pending Jenkins. I'll merge this after tests pass and after I wrap up a final local test with the make-distribution.sh script.

I'd like to get these core changes in now in order to unblock other build patches and so that they get more widespread community testing. If necessary, we can address small issues in followup patches.

SparkQA · 2016-04-04T23:34:11Z

Test build #54892 has finished for PR 11796 at commit a419707.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-04-04T23:51:57Z

I also manually ran DistributedSuite in SBT to make sure that the SBT build still works; everything looked fine, so I'm merging now. Thanks!

zsxwing · 2016-04-05T02:06:06Z

How to build Spark after this patch? Just run build/sbt clean assembly but bin/spark-shell doesn't work any more:

Failed to find Spark jars directory (/Users/zsx/workspace/spark/assembly/target/scala-2.10/jars).
You need to build Spark before running this program.

JoshRosen · 2016-04-05T02:18:48Z

@zsxwing, try build/sbt clean package?

vanzin · 2016-04-05T02:58:58Z

Yeah it should be "package" now. I'll send a message to the dev list.

zsxwing · 2016-04-05T04:38:03Z

yeah, package works. Thanks!

hvanhovell reviewed Mar 17, 2016
View reviewed changes

JoshRosen mentioned this pull request Mar 22, 2016

[SPARK-13808][test-maven] Don't build assembly in dev/run-tests #11701

Closed

vanzin force-pushed the SPARK-13579 branch from b5b8dd1 to 6c4ed0a Compare March 23, 2016 02:40

vanzin force-pushed the SPARK-13579 branch from e37886b to 55aec21 Compare March 23, 2016 17:20

vanzin changed the title ~~[SPARK-13579][build][wip] Stop building the main Spark assembly.~~ [SPARK-13579][build][wip][test-maven] Stop building the main Spark assembly. Mar 23, 2016

vanzin changed the title ~~[SPARK-13579][build][wip][test-maven] Stop building the main Spark assembly.~~ [SPARK-13579][build][wip] Stop building the main Spark assembly. Mar 23, 2016

JoshRosen reviewed Mar 31, 2016
View reviewed changes

Merge branch 'master' into SPARK-13579

9275ea6

Merge branch 'master' into SPARK-13579

b5d4292

JoshRosen reviewed Apr 4, 2016
View reviewed changes

Add missing "jars" in path.

a419707

asfgit closed this in 24d7d2e Apr 4, 2016

vanzin deleted the SPARK-13579 branch April 5, 2016 20:24

		@@ -201,24 +201,29 @@ class FileAppenderSuite extends SparkFunSuite with BeforeAndAfter with Logging {

		// Make sure only logging errors

[SPARK-13579][build][test-maven] Stop building the main Spark assembly. #11796

[SPARK-13579][build][test-maven] Stop building the main Spark assembly. #11796

Conversation

vanzin commented Mar 17, 2016

vanzin commented Mar 17, 2016

SparkQA commented Mar 17, 2016

hvanhovell Mar 17, 2016

Choose a reason for hiding this comment

vanzin Mar 18, 2016

Choose a reason for hiding this comment

hvanhovell Mar 23, 2016

Choose a reason for hiding this comment

vanzin Mar 23, 2016

Choose a reason for hiding this comment

SparkQA commented Mar 17, 2016

SparkQA commented Mar 18, 2016

SparkQA commented Mar 18, 2016

SparkQA commented Mar 19, 2016

vanzin commented Mar 21, 2016

SparkQA commented Mar 21, 2016

SparkQA commented Mar 22, 2016

vanzin commented Mar 22, 2016

vanzin commented Mar 22, 2016

SparkQA commented Mar 22, 2016

vanzin commented Mar 22, 2016

vanzin commented Mar 22, 2016

SparkQA commented Mar 23, 2016

vanzin commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

vanzin commented Mar 23, 2016

vanzin commented Mar 23, 2016

vanzin commented Mar 23, 2016

JoshRosen Mar 31, 2016

Choose a reason for hiding this comment

vanzin Mar 31, 2016

Choose a reason for hiding this comment

JoshRosen commented Mar 31, 2016

SparkQA commented Mar 31, 2016

vanzin commented Mar 31, 2016

JoshRosen commented Mar 31, 2016

JoshRosen commented Mar 31, 2016

SparkQA commented Mar 31, 2016

SparkQA commented Mar 31, 2016

vanzin commented Apr 4, 2016

SparkQA commented Apr 4, 2016

JoshRosen Apr 4, 2016

Choose a reason for hiding this comment

JoshRosen commented Apr 4, 2016

SparkQA commented Apr 4, 2016

JoshRosen commented Apr 4, 2016

zsxwing commented Apr 5, 2016

JoshRosen commented Apr 5, 2016

vanzin commented Apr 5, 2016

zsxwing commented Apr 5, 2016