SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

pwendell · 2015-08-02T22:45:52Z

This is just some small glue code to actually make use of the
AMPLAB_JENKINS_BUILD_TOOL switch. As far as I can tell, we actually
don't currently use the Maven support in the tool even though it exists.
This patch switches to Maven when the PR title contains "test-maven".

There are a few small other pieces of cleanup in the patch as well.

This is just some small glue code to actually make use of the AMPLAB_JENKINS_BUILD_TOOL switch. As far as I can tell, we actually don't currently use the Maven support in the tool. This patch switches to Maven when the PR title contains "maven-test". There are a few small other pieces of cleanup in the patch as well.

pwendell · 2015-08-02T22:53:21Z

/cc @JoshRosen @srowen and @rxin for any thoughts on this.

rxin · 2015-08-02T22:56:29Z

Can it be in the pull request description or the comments?

pwendell · 2015-08-02T23:01:00Z

@rxin unfortunately neither of those is supported by the current PRB infrastructure. We only have access to a limited number of contextual variables set by the PRB plugin, and neither the description or comments are available.

pwendell · 2015-08-02T23:03:37Z

@rxin I've also just added support for triggering different hadoop version builds.

SparkQA · 2015-08-02T23:04:36Z

Test build #39457 has finished for PR 7878 at commit a15f53c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-02T23:07:53Z

Test build #39454 has finished for PR 7878 at commit 6184a00.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaKMeansExample
- class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
- class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
- class SpecificSafeProjection extends $
- case class FromUTCTimestamp(left: Expression, right: Expression)
- case class ToUTCTimestamp(left: Expression, right: Expression)
- case class DateDiff(endDate: Expression, startDate: Expression)
- case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- public final class UnsafeKVExternalSorter

SparkQA · 2015-08-02T23:18:46Z

Test build #39460 has finished for PR 7878 at commit e83b49f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-02T23:20:09Z

Test build #39464 has finished for PR 7878 at commit 5d15d31.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaKMeansExample
- class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
- class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
- class SpecificSafeProjection extends $
- case class FromUTCTimestamp(left: Expression, right: Expression)
- case class ToUTCTimestamp(left: Expression, right: Expression)
- case class DateDiff(endDate: Expression, startDate: Expression)
- case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- public final class UnsafeKVExternalSorter

JoshRosen · 2015-08-02T23:24:40Z

dev/run-tests.py

@@ -231,7 +231,7 @@ def exec_maven(mvn_args=()):
    """Will call Maven in the current directory with the list of mvn_args passed
    in and returns the subprocess for any further processing"""

-    run_cmd([os.path.join(SPARK_HOME, "build", "mvn")] + mvn_args)
+    run_cmd([os.path.join(SPARK_HOME, "build", "mvn")] + ["--force"] + mvn_args)


Technically, the --force can be in the first list, but it doesn't really matter.

JoshRosen · 2015-08-02T23:28:24Z

Just so that you're aware: the diff-dependent test selection logic is not currently implemented for Maven, so pull requests which specify Maven testing will cause all tests to be run.

pwendell · 2015-08-02T23:30:45Z

Jenkins, retest this please.

JoshRosen · 2015-08-02T23:37:12Z

dev/run-tests.py

+
+    cmd = ("/usr/sbin/lsof -P |grep %s | grep LISTEN "
+           "| awk '{ print $2; }' | xargs kill") % zinc_port
+    # TODO: Not sure what happens here if no process exists


On a non-zero exit code, run_cmd will print an error message and will call sys.exit(), so we may want to ensure that this always returns a zero exit code. If this is a concern, you could always just perform the underlying subprocess call yourself.

pwendell · 2015-08-02T23:41:08Z

Jenkins, retest this please.

SparkQA · 2015-08-02T23:47:31Z

Test build #39471 has finished for PR 7878 at commit 69dd9af.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-02T23:50:09Z

Jenkins, retest this please.

SparkQA · 2015-08-02T23:58:06Z

Test build #39468 has finished for PR 7878 at commit c77f951.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-03T00:02:54Z

Jenkins, test this please.

SparkQA · 2015-08-03T00:09:42Z

Test build #39475 has finished for PR 7878 at commit 0d3944a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-03T00:24:03Z

Test build #39476 has finished for PR 7878 at commit 05d7720.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
- class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
- public final class UnsafeKVExternalSorter

pwendell · 2015-08-03T00:25:31Z

Jenkins, retest this please.

SparkQA · 2015-08-03T03:27:52Z

Test build #39478 timed out for PR 7878 at commit 3d36d0a after a configured wait of 175m.

SparkQA · 2015-08-03T07:07:08Z

Test build #39511 has finished for PR 7878 at commit 2e9edcc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-08-03T15:10:53Z

Jenkins, retest this please.

SparkQA · 2015-08-03T17:29:36Z

Test build #39551 has finished for PR 7878 at commit 2e9edcc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-03T21:41:00Z

Jenkins, retest this please.

SparkQA · 2015-08-04T00:07:38Z

Test build #39590 has finished for PR 7878 at commit 2e9edcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Conflicts: dev/run-tests.py

SparkQA · 2015-08-04T01:25:02Z

Test build #39623 has finished for PR 7878 at commit 603f891.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-04T05:27:38Z

Test build #39645 has finished for PR 7878 at commit 87d4d19.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-12T04:25:51Z

Jenkins, test this please.

Conflicts: dev/run-tests-jenkins

pwendell · 2015-08-27T23:25:40Z

Jenkins, test this please.

SparkQA · 2015-08-28T02:10:16Z

Test build #41711 has finished for PR 7878 at commit cf58c49.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-28T04:52:06Z

Jenkins, test this please.

SparkQA · 2015-08-28T08:11:02Z

Test build #41726 has finished for PR 7878 at commit cf58c49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-28T19:45:25Z

Jenkins, test this please.

pwendell · 2015-08-28T19:46:18Z

Okay this seems to be passing now - any thoughts @JoshRosen or @vanzin? IMO this would be really nice since we can test build changes with either build, before they make it into spark.

SparkQA · 2015-08-28T23:03:53Z

Test build #41756 has finished for PR 7878 at commit cf58c49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-28T23:22:47Z

I think this is a good initial step; like Reynold, I'd prefer if this was read from the PR's description. Perhaps the code can be amended later to get the description using the github API and parsing it (and then you can get fancy with how to declare test parameters too).

Also, it seems like the script to merge prs won't clean up these test directives, right? That's more stuff to remember when merging PRs...

On the nits side:

your description says "maven-test" while the directive is "test-maven"
the actual code seems to match "test-maven", not "[test-maven]" as the PR title suggests.

But overall I'm fine with it, we can keep on improving this in later PRs.

vanzin · 2015-08-28T23:23:59Z

dev/run-tests.py

+    """
+    Get a randomized port on which to start Zinc
+    """
+    return random.randrange(3030, 4030)


I hope this doesn't lead to builds failing because zinc can't bind to the chosen port.

This logic is identical to that hard coded in the bash scripts that run the maven builds, in the past they've never failed for this reason... but yeah it could happen.

pwendell · 2015-08-31T04:38:10Z

Thanks for looking at this @vanzin. I do agree it would be a lot nicer to base things on comments, but because the comment stream isn't available as meta-data on jenkins, that's a huge amount of additional work that IMO is best left to an extension if someone is feeling interested. Given that it seems like you are cool to merge this for now as is, then look at evolving it more later.

pwendell changed the title ~~SPARK-9545: Use Maven in PRB if title contains "[maven-test]"~~ SPARK-9545: Use Maven in PRB if title contains "[test-maven]" Aug 2, 2015

Wording change to test-maven

a15f53c

Adding support for SPARK-9547

e83b49f

pwendell changed the title ~~SPARK-9545: Use Maven in PRB if title contains "[test-maven]"~~ SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" Aug 2, 2015

Adding randomized zinc ports

5d15d31

JoshRosen reviewed Aug 2, 2015
View reviewed changes

Fixing up python lint issues

7f1ff75

Small style fix

c77f951

JoshRosen reviewed Aug 2, 2015
View reviewed changes

Push Zinc support down into exec_maven

69dd9af

Small fixes

0d3944a

Bug fix to kill command

05d7720

Increasing timeout

2e9edcc

pwendell added 3 commits August 3, 2015 18:16

Merge remote-tracking branch 'apache/master' into maven-tests

acf1e4f

Conflicts: dev/run-tests.py

Only do compatibility tests in sbt

a938de3

removing unncessary spacing changes

603f891

Whitepsace issues

87d4d19

JoshRosen mentioned this pull request Aug 4, 2015

[SPARK-9607] [SPARK-9608] fix zinc-port handling in build/mvn #7944

Closed

Merge remote-tracking branch 'apache/master' into maven-tests

cf58c49

Conflicts: dev/run-tests-jenkins

vanzin reviewed Aug 28, 2015
View reviewed changes

asfgit closed this in 35e896a Aug 31, 2015

JoshRosen mentioned this pull request Oct 18, 2015

[WIP][SPARK-7018][Build]: Refactor dev/run-tests-jenkins into Python #7401

Closed

JoshRosen added a commit to JoshRosen/spark that referenced this pull request Oct 18, 2015

Port changes from apache#7878.

27f215d

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

Conversation

pwendell commented Aug 2, 2015

pwendell commented Aug 2, 2015

rxin commented Aug 2, 2015

pwendell commented Aug 2, 2015

pwendell commented Aug 2, 2015

SparkQA commented Aug 2, 2015

SparkQA commented Aug 2, 2015

SparkQA commented Aug 2, 2015

SparkQA commented Aug 2, 2015

JoshRosen Aug 2, 2015

Choose a reason for hiding this comment

JoshRosen commented Aug 2, 2015

pwendell commented Aug 2, 2015

JoshRosen Aug 2, 2015

Choose a reason for hiding this comment

pwendell commented Aug 2, 2015

SparkQA commented Aug 2, 2015

pwendell commented Aug 2, 2015

SparkQA commented Aug 2, 2015

pwendell commented Aug 3, 2015

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

pwendell commented Aug 3, 2015

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

JoshRosen commented Aug 3, 2015

SparkQA commented Aug 3, 2015

pwendell commented Aug 3, 2015

SparkQA commented Aug 4, 2015

SparkQA commented Aug 4, 2015

SparkQA commented Aug 4, 2015

pwendell commented Aug 12, 2015

pwendell commented Aug 27, 2015

SparkQA commented Aug 28, 2015

pwendell commented Aug 28, 2015

SparkQA commented Aug 28, 2015

pwendell commented Aug 28, 2015

pwendell commented Aug 28, 2015

SparkQA commented Aug 28, 2015

vanzin commented Aug 28, 2015

vanzin Aug 28, 2015

Choose a reason for hiding this comment

pwendell Aug 31, 2015

Choose a reason for hiding this comment

pwendell commented Aug 31, 2015