Skip to content

[HUDI-7176] Add file group reader test framework#10263

Merged
yihua merged 1 commit intoapache:masterfrom
linliu-code:HUDI-7176-file-group-reader-utils
Dec 12, 2023
Merged

[HUDI-7176] Add file group reader test framework#10263
yihua merged 1 commit intoapache:masterfrom
linliu-code:HUDI-7176-file-group-reader-utils

Conversation

@linliu-code
Copy link
Collaborator

Our goal is to add a framework for unit tests at the level of file group reading. That is,
we don't need to involve any engine specific logic during the read, such that we could

  1. Simplify the test process by providing unified and simple APIs,
  2. Impose more granular read process for testing, and
  3. Improve the test performance since no engine specific logic are involved.

We will add the framework step by step.

Change Logs

Steps:

  1. Add a builder class to construct file group reader.

Impact

  1. Simplify the process to construct file group reader level tests;
  2. Improve the test performance by removing engine specific logic.

Risk level (write none, low medium or high below)

None.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@linliu-code linliu-code changed the title [HUDI-7176] Add file group reader test framework [HUDI-7176][WIP] Add file group reader test framework Dec 6, 2023
@linliu-code linliu-code marked this pull request as draft December 6, 2023 20:05
@linliu-code
Copy link
Collaborator Author

@yihua , @codope , @danny0405

@linliu-code linliu-code force-pushed the HUDI-7176-file-group-reader-utils branch 6 times, most recently from 7ee85e7 to d8ab3f6 Compare December 8, 2023 18:18
@linliu-code linliu-code changed the title [HUDI-7176][WIP] Add file group reader test framework [HUDI-7176] Add file group reader test framework Dec 8, 2023
@linliu-code linliu-code force-pushed the HUDI-7176-file-group-reader-utils branch 2 times, most recently from 9450ab6 to d67c76a Compare December 8, 2023 19:23
Changes:
1. Add a builder class to construct file group reader.
2. Add an indexedRecord based reader context.
3. Implement rest of functions for the file group reader utils.
4. Add a util class for generating FileSlice.
@linliu-code linliu-code force-pushed the HUDI-7176-file-group-reader-utils branch from d67c76a to 15f28aa Compare December 11, 2023 21:57
@linliu-code linliu-code marked this pull request as ready for review December 11, 2023 22:47
@hudi-bot
Copy link
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@linliu-code
Copy link
Collaborator Author

@yihua, I have addressed your comments. My overall plan is to land this PR, and use another PR to further simplify this framework and integrate more with other existing modules. How does it sound?

Copy link
Contributor

@yihua yihua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@yihua yihua merged commit 47ad415 into apache:master Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants