[hotfix] [build] Make sure JAVA_HOME is set for tests. #5441

vanzin · 2015-04-09T20:57:51Z

This is needed at least for YARN integration tests, since $JAVA_HOME is used to launch the executors.

andrewor14 · 2015-04-09T21:24:35Z

LGTM if this works... and I think it will. (edit: actually it doesn't cover SBT tests right?)

JoshRosen · 2015-04-09T21:27:10Z

For SBT, I think we should be able to make a corresponding change in TestSettings:

spark/project/SparkBuild.scala

Line 429 in 470d745

envVars in Test += ("SPARK_DIST_CLASSPATH" ->

vanzin · 2015-04-09T21:27:38Z

Ah, good point, jenkins runs sbt not maven... :-/

JoshRosen · 2015-04-09T21:50:17Z

project/SparkBuild.scala

@@ -119,7 +119,7 @@ object SparkBuild extends PomBuild {
  lazy val publishLocalBoth = TaskKey[Unit]("publish-local", "publish local for m2 and ivy")

  lazy val sharedSettings = graphSettings ++ genjavadocSettings ++ Seq (
-    javaHome   := Properties.envOrNone("JAVA_HOME").map(file),
+    javaHome   := sys.props.get("java.home").map(file),


I'm testing out this patch on one of the Jenkins boxes and it looks like this ended up breaking compilation because java.home points to a JRE, which will cause us to fail with an error when SBT tries to shell out to javac:

java.io.IOException: Cannot run program "/usr/java/jdk1.7.0_71/jre/bin/javac": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at sbt.SimpleProcessBuilder.run(ProcessImpl.scala:349) at sbt.AbstractProcessBuilder.run(ProcessImpl.scala:128) at sbt.AbstractProcessBuilder$$anonfun$runBuffered$1.apply(ProcessImpl.scala:159) at sbt.AbstractProcessBuilder$$anonfun$runBuffered$1.apply(ProcessImpl.scala:159) at sbt.compiler.JavacLogger.buffer(AggressiveCompile.scala:235) at sbt.AbstractProcessBuilder.runBuffered(ProcessImpl.scala:159) at sbt.AbstractProcessBuilder.$bang(ProcessImpl.scala:156) [...]

JoshRosen · 2015-04-09T21:53:36Z

I think the right fix may be to either modify our Spark code to handle the case where JAVA_HOME is undefined and fall back to java.home, or to set JAVA_HOME in our Jenkins environment. I'm going to try to take the latter approach now in order to unblock pull requests. If we want to work on a patch to fix this issue in the long term, I can set up a special pull request builder run which omits / clears JAVA_HOME prior to running tests.

JoshRosen · 2015-04-09T21:55:43Z

I'm modifying all of the builds to set JAVA_HOME=/usr/java/latest.

SparkQA · 2015-04-09T22:01:02Z

Test build #29968 has finished for PR 5441 at commit d71f1bb.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:
- RoaringBitmap-0.4.5.jar
- activation-1.1.jar
- akka-actor_2.10-2.3.4-spark.jar
- akka-remote_2.10-2.3.4-spark.jar
- akka-slf4j_2.10-2.3.4-spark.jar
- aopalliance-1.0.jar
- arpack_combined_all-0.1.jar
- avro-1.7.7.jar
- breeze-macros_2.10-0.11.2.jar
- breeze_2.10-0.11.2.jar
- chill-java-0.5.0.jar
- chill_2.10-0.5.0.jar
- commons-beanutils-1.7.0.jar
- commons-beanutils-core-1.8.0.jar
- commons-cli-1.2.jar
- commons-codec-1.10.jar
- commons-collections-3.2.1.jar
- commons-compress-1.4.1.jar
- commons-configuration-1.6.jar
- commons-digester-1.8.jar
- commons-httpclient-3.1.jar
- commons-io-2.1.jar
- commons-lang-2.5.jar
- commons-lang3-3.3.2.jar
- commons-math-2.1.jar
- commons-math3-3.1.1.jar
- commons-net-2.2.jar
- compress-lzf-1.0.0.jar
- config-1.2.1.jar
- core-1.1.2.jar
- curator-client-2.4.0.jar
- curator-framework-2.4.0.jar
- curator-recipes-2.4.0.jar
- gmbal-api-only-3.0.0-b023.jar
- grizzly-framework-2.1.2.jar
- grizzly-http-2.1.2.jar
- grizzly-http-server-2.1.2.jar
- grizzly-http-servlet-2.1.2.jar
- grizzly-rcm-2.1.2.jar
- groovy-all-2.3.7.jar
- guava-14.0.1.jar
- guice-3.0.jar
- hadoop-annotations-2.2.0.jar
- hadoop-auth-2.2.0.jar
- hadoop-client-2.2.0.jar
- hadoop-common-2.2.0.jar
- hadoop-hdfs-2.2.0.jar
- hadoop-mapreduce-client-app-2.2.0.jar
- hadoop-mapreduce-client-common-2.2.0.jar
- hadoop-mapreduce-client-core-2.2.0.jar
- hadoop-mapreduce-client-jobclient-2.2.0.jar
- hadoop-mapreduce-client-shuffle-2.2.0.jar
- hadoop-yarn-api-2.2.0.jar
- hadoop-yarn-client-2.2.0.jar
- hadoop-yarn-common-2.2.0.jar
- hadoop-yarn-server-common-2.2.0.jar
- ivy-2.4.0.jar
- jackson-annotations-2.4.0.jar
- jackson-core-2.4.4.jar
- jackson-core-asl-1.8.8.jar
- jackson-databind-2.4.4.jar
- jackson-jaxrs-1.8.8.jar
- jackson-mapper-asl-1.8.8.jar
- jackson-module-scala_2.10-2.4.4.jar
- jackson-xc-1.8.8.jar
- jansi-1.4.jar
- javax.inject-1.jar
- javax.servlet-3.0.0.v201112011016.jar
- javax.servlet-3.1.jar
- javax.servlet-api-3.0.1.jar
- jaxb-api-2.2.2.jar
- jaxb-impl-2.2.3-1.jar
- jcl-over-slf4j-1.7.10.jar
- jersey-client-1.9.jar
- jersey-core-1.9.jar
- jersey-grizzly2-1.9.jar
- jersey-guice-1.9.jar
- jersey-json-1.9.jar
- jersey-server-1.9.jar
- jersey-test-framework-core-1.9.jar
- jersey-test-framework-grizzly2-1.9.jar
- jets3t-0.7.1.jar
- jettison-1.1.jar
- jetty-util-6.1.26.jar
- jline-0.9.94.jar
- jline-2.10.4.jar
- jodd-core-3.6.3.jar
- json4s-ast_2.10-3.2.10.jar
- json4s-core_2.10-3.2.10.jar
- json4s-jackson_2.10-3.2.10.jar
- jsr305-1.3.9.jar
- jtransforms-2.4.0.jar
- jul-to-slf4j-1.7.10.jar
- kryo-2.21.jar
- log4j-1.2.17.jar
- lz4-1.2.0.jar
- management-api-3.0.0-b012.jar
- mesos-0.21.0-shaded-protobuf.jar
- metrics-core-3.1.0.jar
- metrics-graphite-3.1.0.jar
- metrics-json-3.1.0.jar
- metrics-jvm-3.1.0.jar
- minlog-1.2.jar
- netty-3.8.0.Final.jar
- netty-all-4.0.23.Final.jar
- objenesis-1.2.jar
- opencsv-2.3.jar
- oro-2.0.8.jar
- paranamer-2.6.jar
- parquet-column-1.6.0rc3.jar
- parquet-common-1.6.0rc3.jar
- parquet-encoding-1.6.0rc3.jar
- parquet-format-2.2.0-rc1.jar
- parquet-generator-1.6.0rc3.jar
- parquet-hadoop-1.6.0rc3.jar
- parquet-jackson-1.6.0rc3.jar
- protobuf-java-2.4.1.jar
- protobuf-java-2.5.0-spark.jar
- py4j-0.8.2.1.jar
- pyrolite-2.0.1.jar
- quasiquotes_2.10-2.0.1.jar
- reflectasm-1.07-shaded.jar
- scala-compiler-2.10.4.jar
- scala-library-2.10.4.jar
- scala-reflect-2.10.4.jar
- scalap-2.10.4.jar
- scalatest_2.10-2.2.1.jar
- slf4j-api-1.7.10.jar
- slf4j-log4j12-1.7.10.jar
- snappy-java-1.1.1.6.jar
- spark-bagel_2.10-1.4.0-SNAPSHOT.jar
- spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
- spark-core_2.10-1.4.0-SNAPSHOT.jar
- spark-graphx_2.10-1.4.0-SNAPSHOT.jar
- spark-launcher_2.10-1.4.0-SNAPSHOT.jar
- spark-mllib_2.10-1.4.0-SNAPSHOT.jar
- spark-network-common_2.10-1.4.0-SNAPSHOT.jar
- spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
- spark-repl_2.10-1.4.0-SNAPSHOT.jar
- spark-sql_2.10-1.4.0-SNAPSHOT.jar
- spark-streaming_2.10-1.4.0-SNAPSHOT.jar
- spire-macros_2.10-0.7.4.jar
- spire_2.10-0.7.4.jar
- stax-api-1.0.1.jar
- stream-2.7.0.jar
- tachyon-0.5.0.jar
- tachyon-client-0.5.0.jar
- uncommons-maths-1.2.2a.jar
- unused-1.0.0.jar
- xmlenc-0.52.jar
- xz-1.0.jar
- zookeeper-3.4.5.jar

vanzin · 2015-04-09T22:11:38Z

@JoshRosen hopefully with the latest patch things should work without JAVA_HOME. Did you make the change already?

JoshRosen · 2015-04-09T22:14:37Z

@vanzin, I pushed that change already but what I can do is configure "new" SparkPullRequestBuilder to not include that change and have it test your PR. One sec...

SparkQA · 2015-04-09T22:22:55Z

Test build #29964 has finished for PR 5441 at commit 6bda399.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-09T22:48:54Z

Test build #657 has finished for PR 5441 at commit 3eeec30.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

vanzin · 2015-04-09T23:03:37Z

BTW I tested this change locally and the YARN tests pass. Command line:

(unset JAVA_HOME; ./build/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0)

I also checked the maven side using the help:evaluate goal to check properties (didn't actually run the tests).

vanzin · 2015-04-09T23:04:40Z

@JoshRosen if I retest this will it pick up the custom builder (without JAVA_HOME)?

vanzin · 2015-04-09T23:43:06Z

Jenkins, retest this please.

JoshRosen · 2015-04-09T23:54:29Z

@vanzin, "NewSparkPullRequestBuilder" is triggered via the http://spark-prs.appspot.com platform. I've granted you the required permissions on that site, so if you "Sign In with GitHub" on that site you should now see a "test with Jenkins" button next to PRs, which triggers the new PRB. I'll trigger a new run now.

SparkQA · 2015-04-09T23:56:18Z

Test build #29973 has finished for PR 5441 at commit 3eeec30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-10T01:33:13Z

Test build #29984 has finished for PR 5441 at commit 3eeec30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-10T01:43:06Z

Test build #659 has finished for PR 5441 at commit 3eeec30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

vanzin · 2015-04-10T02:19:24Z

Ah, success.

vanzin · 2015-04-10T02:21:05Z

Let me know if you'd prefer to have a bug for this instead of [hotfix].

srowen · 2015-04-10T12:20:58Z

Just to see if I have this right: this sets JAVA_HOME to the value of test.java.home. That in turn takes the value of the standard property java.home -- unless JAVA_HOME was set in the env, in which case the profile triggers and test.java.home becomes JAVA_HOME's value. And then the code also changes to check both of these sources anyway. FWIW LGTM

vanzin · 2015-04-10T17:10:19Z

yes, that's it; as I said earlier, maven makes this unnecessarily complicated. :-/

PRs Merged 1. [Internal] Add AppleAwsClientFactory for Mascot (apache#577) 2. Hive: Log new metadata location in commit (apache#4681) 3. change timeout to 120 for now (apache#661) 4. Internal: Add hive_catalog parameter to SparkCatalog (apache#670) 5. Internal: Pull catalog setting to CachedClientPool (apache#673) 6. Core: Defer reading Avro metadata until ManifestFile is read (apache#5206) 7. API: Fix ID assignment in schema merging (apache#5395) 8. AWS: S3OutputStream - failure to close should persist on subsequent close calls (apache#5311) 9. API: Allow schema updates to find fields with case-insensitivity (apache#5440) 10. Spark 3.3: Spark mergeSchema to respect Spark Case Sensitivity Configuration (apache#5441)

WIP: Testing to see whether this fixes the yarn test issue.

6bda399

vanzin mentioned this pull request Apr 9, 2015

[SPARK-4705] Handle multiple app attempts event logs, history server. #5432

Closed

And sbt too.

d71f1bb

JoshRosen reviewed Apr 9, 2015
View reviewed changes

Use JAVA_HOME when available, java.home otherwise.

3eeec30

vanzin changed the title ~~WIP: Testing to see whether this fixes the yarn test issue.~~ [hotfix] [build] Make sure JAVA_HOME is set for tests. Apr 10, 2015

asfgit closed this in 694aef0 Apr 11, 2015

vanzin deleted the yarn-test-test branch April 14, 2015 20:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hotfix] [build] Make sure JAVA_HOME is set for tests. #5441

[hotfix] [build] Make sure JAVA_HOME is set for tests. #5441

vanzin commented Apr 9, 2015

andrewor14 commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen Apr 9, 2015

JoshRosen commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

vanzin commented Apr 9, 2015

vanzin commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

vanzin commented Apr 10, 2015

vanzin commented Apr 10, 2015

srowen commented Apr 10, 2015

vanzin commented Apr 10, 2015

[hotfix] [build] Make sure JAVA_HOME is set for tests. #5441

[hotfix] [build] Make sure JAVA_HOME is set for tests. #5441

Conversation

vanzin commented Apr 9, 2015

andrewor14 commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen Apr 9, 2015

Choose a reason for hiding this comment

JoshRosen commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

vanzin commented Apr 9, 2015

vanzin commented Apr 9, 2015

vanzin commented Apr 9, 2015

JoshRosen commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

vanzin commented Apr 10, 2015

vanzin commented Apr 10, 2015

srowen commented Apr 10, 2015

vanzin commented Apr 10, 2015