[HUDI-7176] Add file group reader test framework#10263
Merged
yihua merged 1 commit intoapache:masterfrom Dec 12, 2023
Merged
Conversation
Collaborator
Author
|
@yihua , @codope , @danny0405 |
7ee85e7 to
d8ab3f6
Compare
jonvex
requested changes
Dec 8, 2023
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Outdated
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Outdated
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Outdated
Show resolved
Hide resolved
9450ab6 to
d67c76a
Compare
Changes: 1. Add a builder class to construct file group reader. 2. Add an indexedRecord based reader context. 3. Implement rest of functions for the file group reader utils. 4. Add a util class for generating FileSlice.
d67c76a to
15f28aa
Compare
yihua
reviewed
Dec 12, 2023
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieTestReaderContext.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
Show resolved
Hide resolved
Collaborator
Author
|
@yihua, I have addressed your comments. My overall plan is to land this PR, and use another PR to further simplify this framework and integrate more with other existing modules. How does it sound? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Our goal is to add a framework for unit tests at the level of file group reading. That is,
we don't need to involve any engine specific logic during the read, such that we could
We will add the framework step by step.
Change Logs
Steps:
Impact
Risk level (write none, low medium or high below)
None.
Contributor's checklist