[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

liancheng · 2014-07-26T03:50:16Z

(This is a replacement of #1399, trying to fix potential HiveThriftServer2 port collision between parallel builds. Please refer to these comments for details.)

JIRA issue: SPARK-2410

Merging the Hive Thrift/JDBC server from branch-1.0-jdbc.

Thanks @chenghao-intel for his initial contribution of the Spark SQL CLI.

…rver profile

Any options in the application option list with the same option name that SparkSubmitArguments recognizes (e.g., --help) are stolen by SparkSubmit instead of passed to the application.

All options behind primary resource are (and should be) recognized as application options now.

…el builds

SparkQA · 2014-07-26T03:53:33Z

QA tests have started for PR 1600. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17215/consoleFull

SparkQA · 2014-07-26T05:07:38Z

QA results for PR 1600:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17215/consoleFull

liancheng · 2014-07-26T06:01:40Z

@marmbrus The build failure was caused by PySpark, please help re-test this, thanks!

concretevitamin · 2014-07-26T06:03:42Z

Jenkins, retest this please.

SparkQA · 2014-07-26T06:08:34Z

QA tests have started for PR 1600. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17217/consoleFull

SparkQA · 2014-07-26T07:24:58Z

QA results for PR 1600:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17217/consoleFull

pwendell · 2014-07-27T20:03:24Z

Okay - lets try this again. I can merge it!

liancheng · 2014-07-28T01:22:38Z

Thanks @pwendell, just checked all failed Jenkins builds ever since then, fortunately none was caused by this PR (phew...).

marmbrus · 2014-07-28T01:24:55Z

What about this one: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17259/consoleFull

liancheng · 2014-07-28T02:00:09Z

Ouch... This is really embarrassing... OK, I'll try to refactor those possibly flaky test code inherited from Shark (e.g. hard coded timeout, etc.).

… fix) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar. In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits: 629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server

@pwendell

(This is a replacement of apache#1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](apache#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes apache#1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

… fix) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar. In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits: 629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server

### What changes were proposed in this pull request? This PR cherry-picks Iceberg fixes to our row-level implementation. apache/iceberg@3bc2f97 apache/iceberg@d5411d2 ### Why are the changes needed? These changes are needed to respect attribute nullability in MERGE. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Iceberg tests.

…he#1600)" This reverts commit 774f8b1.

liancheng added 19 commits July 25, 2014 07:06

Cherry picked the Hive Thrift server

2c4c539

Starts Hive Thrift server via spark-submit

61f39f4

Make HiveThriftServer2 play well with spark-submit

a5310d1

Starts spark-sql shell with spark-submit

3ad4e75

Updated docs for Hive compatibility and Shark migration guide draft

f975d22

Fixed minor issues in spark-sql and start-thriftserver.sh

b8905ba

Added missing license headers

e214aab

Fixed more license header issues

40bafef

Adapts test suites to spark-submit settings

7755062

Addressed all comments by @pwendell

061880f

Updated documents and build scripts for the newly added hive-thriftse…

cfcf461

…rver profile

Starts beeline with spark-submit

9cc0f06

Fixed spark-submit application options handling logic

7db82a1

Any options in the application option list with the same option name that SparkSubmitArguments recognizes (e.g., --help) are stolen by SparkSubmit instead of passed to the application.

Fixed failed test suites

1083e9d

Disabled MIMA for hive-thriftserver

199e3fb

Reordered spark-submit options in spark-shell[.cmd]

fe0af31

All options behind primary resource are (and should be) recognized as application options now.

Updated Spark SQL programming guide docs

21c6cf4

Revert changes related to SPARK-2678, decided to move them to another PR

090beea

Uses random port for HiveThriftServer2 to avoid collision with parall…

ac4618b

…el builds

liancheng mentioned this pull request Jul 26, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

Closed

2 tasks

asfgit closed this in f6ff2a6 Jul 27, 2014

liancheng mentioned this pull request Jul 28, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) #1620

Closed

liancheng deleted the jdbc branch September 24, 2014 00:09

sunchao pushed a commit to sunchao/spark that referenced this pull request Jun 2, 2023

Revert "rdar://101429519 (Fix nullability propagation in MERGE) (apac…

7612abb

…he#1600)" This reverts commit 774f8b1.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

liancheng commented Jul 26, 2014

SparkQA commented Jul 26, 2014

SparkQA commented Jul 26, 2014

liancheng commented Jul 26, 2014

concretevitamin commented Jul 26, 2014

SparkQA commented Jul 26, 2014

SparkQA commented Jul 26, 2014

pwendell commented Jul 27, 2014

liancheng commented Jul 28, 2014

marmbrus commented Jul 28, 2014

liancheng commented Jul 28, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

Conversation

liancheng commented Jul 26, 2014

SparkQA commented Jul 26, 2014

SparkQA commented Jul 26, 2014

liancheng commented Jul 26, 2014

concretevitamin commented Jul 26, 2014

SparkQA commented Jul 26, 2014

SparkQA commented Jul 26, 2014

pwendell commented Jul 27, 2014

liancheng commented Jul 28, 2014

marmbrus commented Jul 28, 2014

liancheng commented Jul 28, 2014