[SPARK-13808][test-maven] Don't build assembly in dev/run-tests #11701

JoshRosen · 2016-03-14T19:36:50Z

As of SPARK-9284 we should no longer need to build the full Spark assembly JAR in order to run tests. Therefore, we should remove the assembly step from dev/run-tests in order to reduce build + test time.

Most of the changes in this PR were originally part of #11178.

JoshRosen · 2016-03-14T21:29:59Z

/cc @vanzin, this patch removes the need to build an assembly before running tests.

vanzin · 2016-03-14T21:42:43Z

dev/run-tests.py

@@ -323,7 +323,7 @@ def get_hadoop_profiles(hadoop_version):
 def build_spark_maven(hadoop_version):
    # Enable all of the profiles for the build:
    build_profiles = get_hadoop_profiles(hadoop_version) + modules.root.build_profile_flags
-    mvn_goals = ["clean", "package", "-DskipTests"]
+    mvn_goals = ["clean", "package", "-DskipTests", "-pl", "!assembly"]


If you're looking at speeding up the build, building and testing in one shot might be a good thing to do at least for maven. Using -fae would allow the build to go as far as it can when something fails.

No need to do that in this change, though.

vanzin · 2016-03-14T21:46:34Z

LGTM if tests pass; although a have a comment about the pyspark tests workaround.

This reverts commit 267aaf9.

SparkQA · 2016-03-14T21:56:42Z

Test build #53094 has finished for PR 11701 at commit 2c10193.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T22:09:31Z

Test build #53120 has finished for PR 11701 at commit 1154eb4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T22:15:55Z

Test build #53110 has finished for PR 11701 at commit 267aaf9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-14T23:14:54Z

Jenkins, retest this please.

SparkQA · 2016-03-15T01:14:09Z

Test build #53133 has finished for PR 11701 at commit 1154eb4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-15T02:22:54Z

Jenkins, retest this please.

SparkQA · 2016-03-15T04:12:40Z

Test build #53148 has finished for PR 11701 at commit 1154eb4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-15T08:52:28Z

Test build #53158 has finished for PR 11701 at commit 1fd489c.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-15T18:44:35Z

Jenkins, retest this please.

SparkQA · 2016-03-15T20:52:48Z

Test build #53209 has finished for PR 11701 at commit 1fd489c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-16T06:33:13Z

Jenkins, retest this please.

JoshRosen · 2016-03-16T06:33:33Z

(Pretty sure that something is wrong here, but hoping that maybe it's just flaky; I'll dig in more when I have time.)

SparkQA · 2016-03-16T08:35:47Z

Test build #53285 has finished for PR 11701 at commit 1fd489c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-16T17:59:45Z

Jenkins, retest this please.

JoshRosen · 2016-03-16T17:59:58Z

(Going to quickly make sure this also works with Maven tests, just to sanity-check)

SparkQA · 2016-03-16T18:18:32Z

Test build #53332 has finished for PR 11701 at commit 1fd489c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-16T18:22:47Z

retest this please

SparkQA · 2016-03-16T18:43:34Z

Test build #53333 has finished for PR 11701 at commit 1fd489c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-16T18:46:00Z

16/03/16 11:43:23.870 SparkLauncherSuite-2 INFO OutputRedirector: Failed to find Spark jars directory (/home/jenkins/workspace/SparkPullRequestBuilder@2/assembly/target/scala-2.10).
16/03/16 11:43:23.870 SparkLauncherSuite-2 INFO OutputRedirector: You need to build Spark before running this program.

Looks like some adjustment is needed in my previous change?

vanzin · 2016-03-16T18:48:02Z

Perhaps changing this in bin/spark-class:

if [ ! -d "$SPARK_JARS_DIR" ] && [ -z "$SPARK_TESTING" ] ; then
  echo "Failed to find Spark jars directory ($SPARK_JARS_DIR)." 1>&2
  echo "You need to build Spark before running this program." 1>&2
  exit 1
fi

SparkQA · 2016-03-16T19:48:04Z

Test build #53336 has finished for PR 11701 at commit 20d2e2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-16T19:57:52Z

bin/spark-class

 fi

-LAUNCH_CLASSPATH="$SPARK_JARS_DIR/*"
-
 # Add the launcher build dir to the classpath if requested.
 if [ -n "$SPARK_PREPEND_CLASSES" ]; then
  LAUNCH_CLASSPATH="${SPARK_HOME}/launcher/target/scala-$SPARK_SCALA_VERSION/classes:$LAUNCH_CLASSPATH"


Somehow this is not being picked up (test is failing because the Main class is not found). Is this path correct for maven too?

vanzin · 2016-03-16T20:22:03Z

Hmmm. I think this is because this code in load-spark-env.sh:

if [ -z "$SPARK_SCALA_VERSION" ]; then
  USER_SCALA_VERSION_SET=0
  ASSEMBLY_DIR2="${SPARK_HOME}/assembly/target/scala-2.11"
  ASSEMBLY_DIR1="${SPARK_HOME}/assembly/target/scala-2.10"

  if [[ -d "$ASSEMBLY_DIR2" && -d "$ASSEMBLY_DIR1" ]]; then
    echo -e "Presence of build for both scala versions(SCALA 2.10 and SCALA 2.11) detected." 1>&2
    echo -e 'Either clean one of them or, export SPARK_SCALA_VERSION=2.11 in spark-env.sh.' 1>&2
    exit 1
  fi

  if [ -d "$ASSEMBLY_DIR2" ]; then
    export SPARK_SCALA_VERSION="2.11"
  else
    export SPARK_SCALA_VERSION="2.10"
  fi
else
    USER_SCALA_VERSION_SET=1
fi

Since the assembly build is being skipped, it's defaulting to scala 2.10 and failing to find the classes.

vanzin · 2016-03-16T20:24:31Z

BTW, I'm just waiting for this patch before working on SPARK-13579; so it's probably ok to keep building the assembly in the maven build, because it will become a lot cheaper once I implement that change (basically just copy files around).

JoshRosen · 2016-03-16T20:26:18Z

Yeah, I literally just found the same issue with the defaults in load-spark-env.sh. I wonder whether it would make sense to just have Maven set that variable appropriately based on the profile.

JoshRosen · 2016-03-16T20:30:07Z

Since all of this code is going to be changed heavily / removed after your final patch, I'm going to go ahead and just leave the Maven test path unchanged so that we can get this merged.

JoshRosen · 2016-03-16T20:32:43Z

python/run-tests.py

+    original_working_dir = os.getcwd()
+    os.chdir(SPARK_HOME)
+    cp = subprocess_check_output(
+        ["./build/sbt", "-Phive", "export assembly/managedClasspath"], universal_newlines=True)


@vanzin, as part of your next patch will SPARK_DIST_CLASSPATH just point to the libs directory? If so, we can remove this as part of that patch.

Potentially. I'll make a note to take a look at this.

JoshRosen · 2016-03-16T22:04:47Z

Jenkins, retest this please.

SparkQA · 2016-03-16T22:21:54Z

Test build #53345 has finished for PR 11701 at commit 0900b13.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-17T00:10:23Z

Test build #53364 has finished for PR 11701 at commit 0900b13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-17T00:17:30Z

Okay, so this passes all Scala tests but is failing PySpark Maven tests...

vanzin · 2016-03-17T17:04:07Z

Are the failures legitimate? When I looked before it seemed only one of the python interpreters failed, which generally screams "flaky test" to me...

JoshRosen · 2016-03-17T18:00:34Z

Jenkins, retest this please.

SparkQA · 2016-03-17T18:13:04Z

Test build #53443 has finished for PR 11701 at commit 0900b13.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-17T18:48:11Z

Jenkins, retest this please.

SparkQA · 2016-03-17T19:38:57Z

Test build #53453 has finished for PR 11701 at commit 0900b13.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-03-22T23:57:46Z

I'm going to close this in favor of @vanzin's #11796

JoshRosen added 2 commits March 14, 2016 12:35

Remove assembly in tests.

2c10193

Fix ClassNotFoundException errors in PySpark streaming tests.

267aaf9

vanzin reviewed Mar 14, 2016
View reviewed changes

JoshRosen added 2 commits March 14, 2016 14:48

Revert "Fix ClassNotFoundException errors in PySpark streaming tests."

e6e3c20

This reverts commit 267aaf9.

Remove external/ projects from SPARK_PREPEND_CLASSES

1154eb4

Enable Hive in PySpark test deps.

1fd489c

JoshRosen changed the title ~~[SPARK-13808] Don't build assembly in dev/run-tests~~ [SPARK-13808][test-maven] Don't build assembly in dev/run-tests Mar 16, 2016

Tolerate missing SPARK_JARS_DIR in tests.

20d2e2a

vanzin reviewed Mar 16, 2016
View reviewed changes

Continue to build assembly in Maven tests (for now)

0900b13

JoshRosen reviewed Mar 16, 2016
View reviewed changes

JoshRosen changed the title ~~[SPARK-13808][test-maven] Don't build assembly in dev/run-tests~~ [SPARK-13808] Don't build assembly in dev/run-tests Mar 16, 2016

JoshRosen changed the title ~~[SPARK-13808] Don't build assembly in dev/run-tests~~ [SPARK-13808][test-maven] Don't build assembly in dev/run-tests Mar 17, 2016

JoshRosen closed this Mar 22, 2016

vanzin pushed a commit to vanzin/spark that referenced this pull request Mar 25, 2016

Fix tests (contains fix borrowed from apache#11701).

597e6ec

vanzin pushed a commit to vanzin/spark that referenced this pull request Mar 25, 2016

Apply some cleanup / fixes from apache#11701.

ef56b81

JoshRosen deleted the remove-assembly-in-tests branch August 29, 2016 19:20

[SPARK-13808][test-maven] Don't build assembly in dev/run-tests #11701

[SPARK-13808][test-maven] Don't build assembly in dev/run-tests #11701

Conversation

JoshRosen commented Mar 14, 2016

JoshRosen commented Mar 14, 2016

vanzin Mar 14, 2016

Choose a reason for hiding this comment

vanzin commented Mar 14, 2016

SparkQA commented Mar 14, 2016

SparkQA commented Mar 14, 2016

SparkQA commented Mar 14, 2016

JoshRosen commented Mar 14, 2016

SparkQA commented Mar 15, 2016

JoshRosen commented Mar 15, 2016

SparkQA commented Mar 15, 2016

SparkQA commented Mar 15, 2016

JoshRosen commented Mar 15, 2016

SparkQA commented Mar 15, 2016

JoshRosen commented Mar 16, 2016

JoshRosen commented Mar 16, 2016

SparkQA commented Mar 16, 2016

JoshRosen commented Mar 16, 2016

JoshRosen commented Mar 16, 2016

SparkQA commented Mar 16, 2016

vanzin commented Mar 16, 2016

SparkQA commented Mar 16, 2016

vanzin commented Mar 16, 2016

vanzin commented Mar 16, 2016

SparkQA commented Mar 16, 2016

vanzin Mar 16, 2016

Choose a reason for hiding this comment

vanzin commented Mar 16, 2016

vanzin commented Mar 16, 2016

JoshRosen commented Mar 16, 2016

JoshRosen commented Mar 16, 2016

JoshRosen Mar 16, 2016

Choose a reason for hiding this comment

vanzin Mar 16, 2016

Choose a reason for hiding this comment

JoshRosen commented Mar 16, 2016

SparkQA commented Mar 16, 2016

SparkQA commented Mar 17, 2016

JoshRosen commented Mar 17, 2016

vanzin commented Mar 17, 2016

JoshRosen commented Mar 17, 2016

SparkQA commented Mar 17, 2016

JoshRosen commented Mar 17, 2016

SparkQA commented Mar 17, 2016

JoshRosen commented Mar 22, 2016