[HUDI-0001] DO NOT MERGE Ci not working test#9983
Closed
jonvex wants to merge 13 commits intoapache:masterfrom
Closed
[HUDI-0001] DO NOT MERGE Ci not working test#9983jonvex wants to merge 13 commits intoapache:masterfrom
jonvex wants to merge 13 commits intoapache:masterfrom
Conversation
To support incremental quereis, we do the following things: 1. Extends the HoodieFileIndex as HoodieIncremtnalFileIndex, in which we override the listFiles and inputFiles methods to return only the affected files during the specified time range. 2. Build a factory class to build different HadoopFsRelation for different query types.
To do this, what we have done: 1. Duplicated the logic for CDCFileGroupIterator logic. 2. Created HoodieCDCFileIndex, which lists files based on the HoodieFileGroupId, which is embedded into partitionValues of PartitionedFiles. 3. Created HadoopFsRelation based on the table type: COW or MOR on DefaultSource.
…s because df.count() is wrong while df.rdd.count() is correct
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Logs
testassdfa
Impact
sdafdsaf
Risk level (write none, low medium or high below)
none
asdfasd
Documentation Update
asfdfad
Contributor's checklist