[SPARK-9284] [tests] Allow all tests to run without an assembly. #7629

vanzin · 2015-07-23T23:31:32Z

This change aims at speeding up the dev cycle a little bit, by making
sure that all tests behave the same w.r.t. where the code to be tested
is loaded from. Namely, that means that tests don't rely on the assembly
anymore, rather loading all needed classes from the build directories.

The main change is to make sure all build directories (classes and test-classes)
are added to the classpath of child processes when running tests.

YarnClusterSuite required some custom code since the executors are run
differently (i.e. not through the launcher library, like standalone and
Mesos do).

I also found a couple of tests that could leak a SparkContext on failure,
and added code to handle those.

With this patch, it's possible to run the following command from a clean
source directory and have all tests pass:

mvn -Pyarn -Phadoop-2.4 -Phive-thriftserver install

Doing this may cause weird errors when tests are run on maven, depending on the flags used. Instead, expose the needed functionality through methods that do not expose shaded classes.

This change aims at speeding up the dev cycle a little bit, by making sure that all tests behave the same w.r.t. where the code to be tested is loaded from. Namely, that means that tests don't rely on the assembly anymore, rather loading all needed classes from the build directories. The main change is to make sure all build directories (classes and test-classes) are added to the classpath of child processes when running tests. YarnClusterSuite required some custom code since the executors are run differently (i.e. not through the launcher library, like standalone and Mesos do). I also found a couple of tests that could leak a SparkContext on failure, and added code to handle those. With this patch, it's possible to run the following command from a clean source directory and have all tests pass: mvn -Pyarn -Phadoop-2.4 -Phive-thriftserver install

vanzin · 2015-07-23T23:32:04Z

Note I included a separate patch in this PR, since I was hitting that issue every time with these changes. I'll rebase once that patch is committed.

SparkQA · 2015-07-23T23:37:44Z

Test build #38281 has finished for PR 7629 at commit 8d2b0ea.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-24T18:15:32Z

Weird that scalastyle passed locally.

SparkQA · 2015-07-24T20:04:24Z

Test build #38387 has finished for PR 7629 at commit 0f5a5bb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends Serializable
- case class ChangeDecimalPrecision(child: Expression) extends UnaryExpression
- class GenericRow(protected[sql] val values: Array[Any]) extends Row
- class GenericInternalRow(protected[sql] val values: Array[Any]) extends InternalRow
- class GenericInternalRowWithSchema(values: Array[Any], val schema: StructType)
- class GenericMutableRow(val values: Array[Any]) extends MutableRow

SparkQA · 2015-07-24T21:08:02Z

Test build #38393 has finished for PR 7629 at commit 3ec7cd4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends Serializable
- case class ChangeDecimalPrecision(child: Expression) extends UnaryExpression
- class GenericRow(protected[sql] val values: Array[Any]) extends Row
- class GenericInternalRow(protected[sql] val values: Array[Any]) extends InternalRow
- class GenericInternalRowWithSchema(values: Array[Any], val schema: StructType)
- class GenericMutableRow(val values: Array[Any]) extends MutableRow

SparkQA · 2015-07-24T23:26:06Z

Test build #38401 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-25T00:35:40Z

Ok, this is down to that kinesis test that's failing for everybody.

vanzin · 2015-07-27T17:34:31Z

Jenkins, retest this please.

SparkQA · 2015-07-27T19:23:25Z

Test build #119 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-28T17:41:44Z

Jenkins retest this please.

SparkQA · 2015-07-28T19:44:39Z

Test build #138 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-28T22:39:54Z

Jenkins, retest this please.

SparkQA · 2015-07-29T00:33:23Z

Test build #140 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-29T00:35:23Z

Test build #38774 has finished for PR 7629 at commit 469a732.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-31T18:30:16Z

retest this please

SparkQA · 2015-07-31T20:37:26Z

Test build #182 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-31T20:48:02Z

Test build #39251 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-07-31T21:55:53Z

retest this please

SparkQA · 2015-08-01T00:00:10Z

Test build #39280 has finished for PR 7629 at commit 469a732.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-01T00:17:23Z

Test build #183 has finished for PR 7629 at commit 469a732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-01T02:20:31Z

Test build #39303 has finished for PR 7629 at commit 8e4a136.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-01T20:15:03Z

retest this please

SparkQA · 2015-08-01T22:03:47Z

Test build #188 has finished for PR 7629 at commit 8e4a136.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-24T23:36:16Z

Test build #41471 timed out for PR 7629 at commit d234399 after a configured wait of 175m.

vanzin · 2015-08-24T23:39:22Z

Seems like all tests passed, no idea why jenkins thinks they timed out.

AFAICT, this is good to do. @pwendell ?

pwendell · 2015-08-25T00:16:15Z

Yeah sounds good - might be good to let it run one more time just to be
sure it's not affecting jenkins somehow.

On Mon, Aug 24, 2015 at 4:40 PM, Marcelo Vanzin notifications@github.com
wrote:

Seems like all tests passed, no idea why jenkins thinks they timed out.

AFAICT, this is good to do. @pwendell https://github.com/pwendell ?

—
Reply to this email directly or view it on GitHub
#7629 (comment).

pwendell · 2015-08-25T00:16:21Z

Jenkins, test this please.

SparkQA · 2015-08-25T03:19:36Z

Test build #41492 timed out for PR 7629 at commit d234399 after a configured wait of 175m.

vanzin · 2015-08-25T17:43:09Z

175m is starting to look really low. the scala/java unit tests took 143m to run. anyway, retest this please.

pwendell · 2015-08-25T17:45:02Z

Does this PR increase test time in some way? Just wondering why this would
consistently timeout when others don't.

On Tue, Aug 25, 2015 at 10:43 AM, Marcelo Vanzin notifications@github.com
wrote:

175m is starting to look really low. the scala/java unit tests took 143m
to run. anyway, retest this please.

—
Reply to this email directly or view it on GitHub
#7629 (comment).

vanzin · 2015-08-25T17:47:38Z

No, it shouldn't affect the test runtime at all. And I get timeouts in lots of PRs, not just this one.

SparkQA · 2015-08-25T20:33:09Z

Test build #41539 has finished for PR 7629 at commit d234399.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-25T20:35:38Z

Jenkins, retest this please.

SparkQA · 2015-08-25T23:19:51Z

Test build #41559 has finished for PR 7629 at commit d234399.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-26T02:59:03Z

Test build #41574 has finished for PR 7629 at commit a2c7c59.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-26T03:17:47Z

I've seen the failing test fail in other PRs (and it has passed here before)... let's try again just in case. retest this please

SparkQA · 2015-08-26T06:23:29Z

Test build #41584 timed out for PR 7629 at commit a2c7c59 after a configured wait of 175m.

pwendell · 2015-08-26T06:56:56Z

Marcelo you will need to change the timeout in the code itself for it to
increase from 175
On Aug 25, 2015 11:24 PM, "UCB AMPLab" notifications@github.com wrote:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41584/
Test FAILed.

—
Reply to this email directly or view it on GitHub
#7629 (comment).

vanzin · 2015-08-26T16:14:25Z

you will need to change the timeout in the code

Yes but I don't want to do that as part of this change, since they're unrelated things. All tests have been passing, the timeouts are unrelated to the PR.

pwendell · 2015-08-26T16:43:51Z

Okay then - I would do that separately, get this PR passing, then merge it.
It is not good to merge a PR that deterministically fails jenkins. Have we
done that in the recent past? I saw a few other PR's that hit an occasional
timeout, but this one seems to be timing out with certainty every time.

On Wed, Aug 26, 2015 at 9:15 AM, Marcelo Vanzin notifications@github.com
wrote:

you will need to change the timeout in the code

Yes but I don't want to do that as part of this change, since they're
unrelated things. All tests have been passing, the timeouts are unrelated
to the PR.

—
Reply to this email directly or view it on GitHub
#7629 (comment).

vanzin · 2015-08-26T16:46:11Z

but this one seems to be timing out with certainty every time

Often, but not deterministically every time. It seems to time out as often as any other PR that needs to run all tests (I've already cc'ed you on at least another one that times out just as often).

pwendell · 2015-08-26T16:48:28Z

I still don't understand - why not fix the issue in a separate PR, get this
passing and then merge this? It will then benefit the other PR's also that
are facing this issue.

On Wed, Aug 26, 2015 at 9:46 AM, Marcelo Vanzin notifications@github.com
wrote:

but this one seems to be timing out with certainty every time

Often, but not deterministically every time. It seems to time out as often
as any other PR that needs to run all tests (I've already cc'ed you on at
least another one that times out just as often).

—
Reply to this email directly or view it on GitHub
#7629 (comment).

vanzin · 2015-08-26T16:50:22Z

I'm working on fixing the root cause of the timeouts (running unnecessary tests). If you think it would be beneficial to just bump the timeout right now, please just send a PR for that; I'm pretty confident that this PR does not make the timeout issue any worse.

pwendell · 2015-08-26T19:20:06Z

Jenkins, retest this please.

pwendell · 2015-08-26T19:20:25Z

K - I just sent a hotfix to up the timeout.

On Wed, Aug 26, 2015 at 9:51 AM, Marcelo Vanzin notifications@github.com
wrote:

I'm working on fixing the root cause of the timeouts (running unnecessary
tests). If you think it would be beneficial to just bump the timeout right
now, please just send a PR for that; I'm pretty confident that this PR does
not make the timeout issue any worse.

—
Reply to this email directly or view it on GitHub
#7629 (comment).

SparkQA · 2015-08-26T22:26:30Z

Test build #41647 has finished for PR 7629 at commit a2c7c59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-26T22:53:09Z

Yay!

pwendell · 2015-08-26T22:59:33Z

Great - looks good!
On Aug 26, 2015 3:53 PM, "Marcelo Vanzin" notifications@github.com wrote:

Yay!

—
Reply to this email directly or view it on GitHub
#7629 (comment).

vanzin · 2015-08-28T19:33:01Z

Ok I'm merging this.

Marcelo Vanzin added 2 commits July 23, 2015 11:23

[SPARK-9261] [streaming] Avoid calling APIs that expose shaded classes.

78865c3

Doing this may cause weird errors when tests are run on maven, depending on the flags used. Instead, expose the needed functionality through methods that do not expose shaded classes.

Marcelo Vanzin added 2 commits July 24, 2015 10:25

Merge branch 'master' into SPARK-9284

2775a64

Style.

0f5a5bb

Merge branch 'master' into SPARK-9284

3ec7cd4

Remove reference to constant added in newer YARN versions.

469a732

Merge branch 'master' into SPARK-9284

8e4a136

Marcelo Vanzin added 2 commits August 25, 2015 16:53

Merge branch 'master' into SPARK-9284

16ae221

Compilation fix due to unrelated change.

a2c7c59

asfgit closed this in c53c902 Aug 28, 2015

vanzin deleted the SPARK-9284 branch September 9, 2015 23:10

[SPARK-9284] [tests] Allow all tests to run without an assembly. #7629

[SPARK-9284] [tests] Allow all tests to run without an assembly. #7629

Conversation

vanzin commented Jul 23, 2015

vanzin commented Jul 23, 2015

SparkQA commented Jul 23, 2015

vanzin commented Jul 24, 2015

SparkQA commented Jul 24, 2015

SparkQA commented Jul 24, 2015

SparkQA commented Jul 24, 2015

vanzin commented Jul 25, 2015

vanzin commented Jul 27, 2015

SparkQA commented Jul 27, 2015

vanzin commented Jul 28, 2015

SparkQA commented Jul 28, 2015

vanzin commented Jul 28, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 29, 2015

vanzin commented Jul 31, 2015

SparkQA commented Jul 31, 2015

SparkQA commented Jul 31, 2015

vanzin commented Jul 31, 2015

SparkQA commented Aug 1, 2015

SparkQA commented Aug 1, 2015

SparkQA commented Aug 1, 2015

vanzin commented Aug 1, 2015

SparkQA commented Aug 1, 2015

SparkQA commented Aug 24, 2015

vanzin commented Aug 24, 2015

pwendell commented Aug 25, 2015

pwendell commented Aug 25, 2015

SparkQA commented Aug 25, 2015

vanzin commented Aug 25, 2015

pwendell commented Aug 25, 2015

vanzin commented Aug 25, 2015

SparkQA commented Aug 25, 2015

pwendell commented Aug 25, 2015

SparkQA commented Aug 25, 2015

SparkQA commented Aug 26, 2015

vanzin commented Aug 26, 2015

SparkQA commented Aug 26, 2015

pwendell commented Aug 26, 2015

vanzin commented Aug 26, 2015

pwendell commented Aug 26, 2015

vanzin commented Aug 26, 2015

pwendell commented Aug 26, 2015

vanzin commented Aug 26, 2015

pwendell commented Aug 26, 2015

pwendell commented Aug 26, 2015

SparkQA commented Aug 26, 2015

vanzin commented Aug 26, 2015

pwendell commented Aug 26, 2015

vanzin commented Aug 28, 2015