Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests #8155

Closed
wants to merge 11 commits into from
Closed

[SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests #8155

wants to merge 11 commits into from

Conversation

yhuai
Copy link
Contributor

@yhuai yhuai commented Aug 13, 2015

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #40704 has finished for PR 8155 at commit a19f84f.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #40707 has finished for PR 8155 at commit fd9da93.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1526 timed out for PR 8155 at commit 8b2e4a7 after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1537 has started for PR 8155 at commit 8b2e4a7.

@liancheng
Copy link
Contributor

Here's another place where this configuration is changed

spark/sql/core/src/test/scala/org/apache/spark/sql/SQLConfSuite.scala

Line 77 in 84a2791

ctx.sql(s"set ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS}=10")

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #40743 timed out for PR 8155 at commit d09b603 after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1550 has finished for PR 8155 at commit d09b603.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • trait Identifiable
    • class VectorUDT extends UserDefinedType[Vector]

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #40769 has finished for PR 8155 at commit c6f5abe.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1557 timed out for PR 8155 at commit c6f5abe after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1561 timed out for PR 8155 at commit c6f5abe after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1562 timed out for PR 8155 at commit c6f5abe after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 13, 2015

Test build #1563 has finished for PR 8155 at commit c6f5abe.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1572 timed out for PR 8155 at commit c6f5abe after a configured wait of 175m.

@yhuai
Merge remote-tracking branch 'upstream/master' into setNumPartitionsC…
dc11a4d 
…orrectly

Conflicts:
	sql/core/src/test/scala/org/apache/spark/sql/SQLConfSuite.scala
	sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala
@yhuai
style
4d6fcf8
@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1581 timed out for PR 8155 at commit 4e548f5 after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1587 timed out for PR 8155 at commit 4e548f5 after a configured wait of 175m.

@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1591 has finished for PR 8155 at commit 4d6fcf8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • implicit class StringToColumn(val sc: StringContext)

@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1590 has finished for PR 8155 at commit 4e548f5.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 14, 2015

Test build #1592 has finished for PR 8155 at commit 4d6fcf8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • implicit class StringToColumn(val sc: StringContext)

@rxin
Copy link
Contributor

rxin commented Aug 14, 2015

cc @andrewor14 can you review this?

@andrewor14
Copy link
Contributor

Discussed with @yhuai offline. The current approach will not work if the test does a conf.clear. Instead we'll override the default value in the SQLConf.

@yhuai yhuai changed the title [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests [SPARK-9925] [SQL] [TESTS] [WIP] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests Aug 14, 2015
@yhuai
Copy link
Contributor Author

yhuai commented Aug 17, 2015

@andrewor14 Just updated the pr. For TestSQLContext and TestHiveContext, we define a map of overrideConfs, which will be set when we construct the SQLConf as well as when we call conf.clear().

@yhuai yhuai changed the title [SPARK-9925] [SQL] [TESTS] [WIP] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for tests Aug 17, 2015
@SparkQA
Copy link

SparkQA commented Aug 17, 2015

Test build #41028 has finished for PR 8155 at commit 75aeab7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@andrewor14
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Aug 17, 2015

Test build #41033 has finished for PR 8155 at commit 75aeab7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@andrewor14
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Aug 17, 2015

Test build #41041 has finished for PR 8155 at commit 75aeab7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 17, 2015

Test build #41044 has finished for PR 8155 at commit bc4b8e0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Aug 17, 2015

looks like there is a legitimate failure.

@andrewor14
Copy link
Contributor

@yhuai do you have time to update this?

@yhuai
Copy link
Contributor Author

yhuai commented Sep 3, 2015

Will do it tomorrow.

@davies
Copy link
Contributor

davies commented Sep 4, 2015

Since @yhuai is busy with others, I will take over this one.

@andrewor14
Copy link
Contributor

OK, sounds good

@davies davies mentioned this pull request Sep 4, 2015
Closed
@yhuai
Copy link
Contributor Author

yhuai commented Sep 4, 2015

Thanks! I am closing this PR.

@yhuai yhuai closed this Sep 4, 2015
asfgit pushed a commit that referenced this pull request Sep 5, 2015
@yhuai
[SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correct…
47058ca 
…ly for tests

This PR fix the failed test and conflict for #8155

https://issues.apache.org/jira/browse/SPARK-9925

Closes #8155

Author: Yin Huai <yhuai@databricks.com>
Author: Davies Liu <davies@databricks.com>

Closes #8602 from davies/shuffle_partitions.
@liancheng liancheng mentioned this pull request Sep 5, 2015
Closed
asfgit pushed a commit that referenced this pull request Sep 5, 2015
@liancheng @rxin
[HOTFIX] [SQL] Fixes compilation error
6c75194 
Jenkins master builders are currently broken by a merge conflict between PR #8584 and PR #8155.

Author: Cheng Lian <lian@databricks.com>

Closes #8614 from liancheng/hotfix/fix-pr-8155-8584-conflict.
@liancheng liancheng mentioned this pull request Sep 11, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
6 participants
@yhuai @SparkQA @liancheng @rxin @andrewor14 @davies