[HUDI-8551] Support no precombine field with EVENT_TIME_ORDERING merge mode#12292
Merged
codope merged 5 commits intoapache:masterfrom Jan 29, 2025
Merged
[HUDI-8551] Support no precombine field with EVENT_TIME_ORDERING merge mode#12292codope merged 5 commits intoapache:masterfrom
codope merged 5 commits intoapache:masterfrom
Conversation
4 tasks
bd2b267 to
511d02e
Compare
nsivabalan
reviewed
Jan 28, 2025
Contributor
nsivabalan
left a comment
There was a problem hiding this comment.
can we de-couple partial update support w/ Commit time ordering and sql writes separately
Once we do that, we are good to land. changes LGTM otherwise.
511d02e to
5c65a5d
Compare
codope
approved these changes
Jan 29, 2025
Collaborator
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Logs
This PR fixes the issue when the precombine field is not set with the merge mode of
EVENT_TIME_ORDERING. Note that after #12600, there is no default of the precombine field config and it is no longer required forCOMMIT_TIME_ORDERING. ForEVENT_TIME_ORDERING, if the precombine field is not set, the logic should fall back to use the default ordering value of(int) 0which has the highest priority (i.e., commit time ordering).Detailed fixes include:
HoodieSparkRecordandHoodieAvroIndexedRecordto avoid NPE;New tests are added in
TestMORDataSourceandTestHoodieDeltaStreamerto cover the failure cases without this PR.Impact
Supports no precombine field with EVENT_TIME_ORDERING merge mode.
Risk level
low
Documentation Update
Release docs will be updated.
Contributor's checklist