[HUDI-3695] Add a ORC reader in HoodieBaseRelation#5111
[HUDI-3695] Add a ORC reader in HoodieBaseRelation#5111miomiocat wants to merge 1 commit intoapache:masterfrom
Conversation
| filters: Seq[Filter], | ||
| options: Map[String, String], | ||
| hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = { | ||
| val hfileReader = createHFileReader( |
There was a problem hiding this comment.
The reason why it was called here as opposed to call w/in the returned lambda -- is to avoid serialization of all these passed in parameters down to Spark executor. This is likely the reason why your build is failing at the moment.
Let's keep it as is, and please add a comment elaborating why this should be done in here (and executed w/in Spark Driver)
There was a problem hiding this comment.
Thanks for your advise 👍
Modified according to your comments, PTAL
miomiocat
force-pushed
the
orc-reader
branch
2 times, most recently
from
ff29427 to
a9ddb49
Compare
miomiocat
changed the title
|
@miomiocat Add UT Test for this PR? |
|
@alexeykudinkin : can you follow up on the review when you get a chance. |
Thanks for your reply. Rebased, PTAL |
|
hi @miomiocat fix the CI failed. |
fixed, PTAL |
|
@alexeykudinkin @XuQianJin-Stars Could you help to review this PR? |
|
@miomiocat can you please rebase this one? |
Thanks for your reply, but I don't have much time to work on this PR recently. closed it. |
5 participants
What is the purpose of the pull request
Add a orc reader in HoodieBaseRelation
Brief change log
Add a orc reader in HoodieBaseRelation
Verify this pull request
This pull request is already covered by existing tests
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.