[SPARK-13403] [SQL] Pass hadoopConfiguration to HiveConf constructors. #11273
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @marmbrus |
|
Seems reasonable. |
|
ok to test |
|
Test build #51669 has finished for PR 11273 at commit
|
|
FWIW also seems reasonable to me; making a new conf is rarely correct. @rdblue can you rebase? |
|
Also, could you fix the title to follow the Spark convention? |
rdblue
changed the title
sc.hadoopConfiguration may contain Hadoop-specific configuration properties that are not used by SparkSQL's HiveContext because it is not passed when constructing instances of HiveConf.
rdblue
force-pushed
the
SPARK-13403-new-hive-conf-from-hadoop-conf
branch
from
32a3dcf
to
f0f6cee
Compare
|
Test build #53377 has finished for PR 11273 at commit
|
|
Thanks - I'm going to merge this. |
|
Thanks for the reviews! |
roygao94
pushed a commit
to roygao94/spark
that referenced
this pull request
Mar 22, 2016
This commit updates the HiveContext so that sc.hadoopConfiguration is used to instantiate its internal instances of HiveConf. I tested this by overriding the S3 FileSystem implementation from spark-defaults.conf as "spark.hadoop.fs.s3.impl" (to avoid [HADOOP-12810](https://issues.apache.org/jira/browse/HADOOP-12810)). Author: Ryan Blue <blue@apache.org> Closes apache#11273 from rdblue/SPARK-13403-new-hive-conf-from-hadoop-conf.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit updates the HiveContext so that sc.hadoopConfiguration is used to instantiate its internal instances of HiveConf.
I tested this by overriding the S3 FileSystem implementation from spark-defaults.conf as "spark.hadoop.fs.s3.impl" (to avoid HADOOP-12810).