Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) #1620

Closed
wants to merge 2 commits into from
Closed

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) #1620

wants to merge 2 commits into from

Conversation

liancheng
Copy link
Contributor

JIRA issue: SPARK-2410

Another try for #1399 & #1600. Those two PR breaks Jenkins builds because we made a separate profile hive-thriftserver in sub-project assembly, but the hive-thriftserver module is defined outside the hive-thriftserver profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in hive-thriftserver, but tests fail because related .class files are not included in the assembly jar.

In the most recent commit, module hive-thriftserver is moved into its own profile to fix this problem. All previous commits are squashed for clarity.

@liancheng
Cherry picked the Hive Thrift server
ec3c7a7 
Starts Hive Thrift server via spark-submit

Make HiveThriftServer2 play well with spark-submit

Starts spark-sql shell with spark-submit

Updated docs for Hive compatibility and Shark migration guide draft

Fixed minor issues in spark-sql and start-thriftserver.sh

Added missing license headers

Fixed more license header issues

Adapts test suites to spark-submit settings

Addressed all comments by @pwendell

Updated documents and build scripts for the newly added hive-thriftserver profile

Starts beeline with spark-submit

Fixed spark-submit application options handling logic

Any options in the application option list with the same option name
that SparkSubmitArguments recognizes (e.g., --help) are stolen by
SparkSubmit instead of passed to the application.

Fixed failed test suites

Disabled MIMA for hive-thriftserver

Reordered spark-submit options in spark-shell[.cmd]

All options behind primary resource are (and should be) recognized as
application options now.

Updated Spark SQL programming guide docs

Revert changes related to SPARK-2678, decided to move them to another PR

Uses random port for HiveThriftServer2 to avoid collision with parallel builds
@liancheng
Moved hive-thriftserver module definition into its own profile
629988e
@SparkQA
Copy link

SparkQA commented Jul 28, 2014

QA tests have started for PR 1620. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 28, 2014

QA results for PR 1620:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@marmbrus
Copy link
Contributor

Hopefully third times a charm :)

Merged!

@liancheng
Copy link
Contributor Author

Hmm, didn't find related code in master, and this PR is still open, seems it's not successfully merged yet?

@marmbrus
Copy link
Contributor

https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=a7a9d14479ea6421513a962ff0f45cb969368bab

GitHub is lagging... We filled a bug with infra.
On Jul 28, 2014 5:09 PM, "Cheng Lian" notifications@github.com wrote:

Hmm, didn't find related code in master, and this PR is still open, seems
it's not successfully merged yet?

—
Reply to this email directly or view it on GitHub
#1620 (comment).

@liancheng
Copy link
Contributor Author

Oh, really "lucky" these days :)

@asfgit asfgit closed this in a7a9d14 Jul 29, 2014
@liancheng liancheng mentioned this pull request Aug 2, 2014
Closed
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@liancheng @conviva-zz
[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile…
dc90bcc 
… fix)

JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410)

Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar.

In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits:

629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile
ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server
@liancheng liancheng deleted the jdbc-with-maven-fix branch September 24, 2014 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@liancheng @SparkQA @marmbrus