Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-6505][SQL]Remove the reflection call in HiveFunctionWrapper #5660

Closed
wants to merge 4 commits into from

Conversation

baishuo
Copy link
Contributor

@baishuo baishuo commented Apr 23, 2015

according @liancheng‘s comment in https://issues.apache.org/jira/browse/SPARK-6505, this patch remove the reflection call in HiveFunctionWrapper, and implement the functions named "deserializeObjectByKryo" and "serializeObjectByKryo" according the functions with the save name in
org.apache.hadoop.hive.ql.exec.Utilities.java

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@liancheng
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30848 has started for PR 5660 at commit 0b522a7.

@liancheng
Copy link
Contributor

Would like to add some background about this change for future reference:

Spark SQL repackaged Hive jars to remove unnecessary shading and dependencies. One of them is Kryo. Hive shades Kryo to package org.apache.hive.com.esotericsoftware. Normally this should be fine. However, MapR replaces Spark SQL's repackaged Hive dependencies with genuine Hive packages, thus the reflection call mentioned in this PR couldn't find Kryo because it's moved into another package. On the other hand, the two reflected methods are actually quite simple. That's why we chose to reimplement them in Spark SQL instead of using them via reflection.

import java.io.{OutputStream, InputStream}

import com.esotericsoftware.kryo.io.Input
import com.esotericsoftware.kryo.io.Output
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move imports to the header of the file.

@liancheng
Copy link
Contributor

add to whitelist

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30849 has started for PR 5660 at commit 0b522a7.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30848 has finished for PR 5660 at commit 0b522a7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30848/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30849 has finished for PR 5660 at commit 0b522a7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30849/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Apr 24, 2015

Test build #30892 has started for PR 5660 at commit ae61ec4.

@SparkQA
Copy link

SparkQA commented Apr 24, 2015

Test build #30892 has finished for PR 5660 at commit ae61ec4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30892/
Test PASSed.

@baishuo
Copy link
Contributor Author

baishuo commented Apr 26, 2015

can this patch be merged? :)

@liancheng
Copy link
Contributor

Thanks for working on this! I'm merging this to master.

@asfgit asfgit closed this in 82bb7fd Apr 27, 2015
@srowen
Copy link
Member

srowen commented Apr 30, 2015

@baishuo what's your Apache JIRA username? I can assign you to the JIRA to credit you.

@baishuo
Copy link
Contributor Author

baishuo commented May 7, 2015

hi @srowen , my jira name is also baishuo
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=baishuo
thanks :)

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 14, 2015
according liancheng‘s  comment in https://issues.apache.org/jira/browse/SPARK-6505,  this patch remove the  reflection call in HiveFunctionWrapper, and implement the functions named "deserializeObjectByKryo" and "serializeObjectByKryo" according the functions with the save name in
org.apache.hadoop.hive.ql.exec.Utilities.java

Author: baishuo <vc_java@hotmail.com>

Closes apache#5660 from baishuo/SPARK-6505-20150423 and squashes the following commits:

ae61ec4 [baishuo] modify code style
78d9fa3 [baishuo] modify code style
0b522a7 [baishuo] modify code style
a5ff9c7 [baishuo] Remove the reflection call in HiveFunctionWrapper
@liancheng liancheng deleted the SPARK-6505-20150423 branch May 28, 2015 11:37
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
according liancheng‘s  comment in https://issues.apache.org/jira/browse/SPARK-6505,  this patch remove the  reflection call in HiveFunctionWrapper, and implement the functions named "deserializeObjectByKryo" and "serializeObjectByKryo" according the functions with the save name in
org.apache.hadoop.hive.ql.exec.Utilities.java

Author: baishuo <vc_java@hotmail.com>

Closes apache#5660 from baishuo/SPARK-6505-20150423 and squashes the following commits:

ae61ec4 [baishuo] modify code style
78d9fa3 [baishuo] modify code style
0b522a7 [baishuo] modify code style
a5ff9c7 [baishuo] Remove the reflection call in HiveFunctionWrapper
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants