[SPARK-8604] [SQL] HadoopFsRelation subclasses should set their output format class #6998
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -194,6 +194,16 @@ private[sql] class OrcRelation( | |||
| } | |||
|
|
|||
| override def prepareJobForWrite(job: Job): OutputWriterFactory = { | |||
| job.getConfiguration match { | |||
There was a problem hiding this comment.
Why not job.setOutputFormatClass(classOf[OrcNewOutputFormat])?
There was a problem hiding this comment.
OrcNewOutputFormat also works in this case, but in general the current ORC data source is implemented upon the old OrcOutputFormat because we are still using Hive 0.13.1 right now. It would be confusing to mix OrcNewOutputFormat here.
|
Test build #35723 has finished for PR 6998 at commit
|
|
Test build #35729 has finished for PR 6998 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Jun 25, 2015
…t format class `HadoopFsRelation` subclasses, especially `ParquetRelation2` should set its own output format class, so that the default output committer can be setup correctly when doing appending (where we ignore user defined output committers). Author: Cheng Lian <lian@databricks.com> Closes #6998 from liancheng/spark-8604 and squashes the following commits: 9be51d1 [Cheng Lian] Adds more comments 6db1368 [Cheng Lian] HadoopFsRelation subclasses should set their output format class (cherry picked from commit c337844) Signed-off-by: Cheng Lian <lian@databricks.com>
|
Merged to master and branch-1.4. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
HadoopFsRelationsubclasses, especiallyParquetRelation2should set its own output format class, so that the default output committer can be setup correctly when doing appending (where we ignore user defined output committers).