Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-9206] [SQL] Fix HiveContext classloading for GCS connector. #7549

Closed
wants to merge 1 commit into from

Conversation

dennishuo
Copy link
Contributor

IsolatedClientLoader.isSharedClass includes all of com.google., presumably
for Guava, protobuf, and/or other shared Google libraries, but needs to
count com.google.cloud.
as "hive classes" when determining which ClassLoader
to use. Otherwise, things like HiveContext.parquetFile will throw a
ClassCastException when fs.defaultFS is set to a Google Cloud Storage (gs://)
path. On StackOverflow: http://stackoverflow.com/questions/31478955

EDIT: Adding @yhuai who worked on the relevant classloading isolation pieces.

IsolatedClientLoader.isSharedClass includes all of com.google.*, presumably
for Guava, protobuf, and/or other shared Google libraries, but needs to
count com.google.cloud.* as "hive classes" when determining which ClassLoader
to use. Otherwise, things like HiveContext.parquetFile will throw a
ClassCastException when fs.defaultFS is set to a Google Cloud Storage (gs://)
path. On StackOverflow: http://stackoverflow.com/questions/31478955
@marmbrus
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Jul 21, 2015

Test build #37878 has finished for PR 7549 at commit 1f8db07.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • trait Nondeterministic
    • abstract class RDG extends LeafExpression with Nondeterministic
    • case class Rand(seed: Long) extends RDG
    • case class Randn(seed: Long) extends RDG

@marmbrus
Copy link
Contributor

test this please

1 similar comment
@marmbrus
Copy link
Contributor

test this please

@SparkQA
Copy link

SparkQA commented Jul 21, 2015

Test build #37970 has finished for PR 7549 at commit 1f8db07.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

Thanks! Merging to master.

@asfgit asfgit closed this in c07838b Jul 21, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants