[HUDI-2461] Support out of order commits in MDT with completion time view #9871
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danny0405
reviewed
Oct 17, 2023
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java
Outdated
Show resolved
Hide resolved
danny0405
reviewed
Oct 19, 2023
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java
Outdated
Show resolved
Hide resolved
codope
commented
Oct 19, 2023
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
Show resolved
Hide resolved
codope
force-pushed
the
HUDI-2461-mdt-compact
branch
from
f0a1258
to
975e7bc
Compare
danny0405
reviewed
Oct 31, 2023
...ient/hudi-java-client/src/test/java/org/apache/hudi/client/TestJavaHoodieBackedMetadata.java
Show resolved
Hide resolved
danny0405
approved these changes
Nov 1, 2023
danny0405
reviewed
Nov 1, 2023
...hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java
Outdated
Show resolved
Hide resolved
danny0405
requested changes
Nov 1, 2023
codope
force-pushed
the
HUDI-2461-mdt-compact
branch
from
ea5ea6e
to
a6b8579
Compare
…view Use normal instant time for compaction in MDT Revert "Use normal instant time for compaction in MDT" This reverts commit 6176139. Fix some tests Fix more tests
rebase
codope
force-pushed
the
HUDI-2461-mdt-compact
branch
from
a6b8579
to
65517de
Compare
codope
commented
Nov 3, 2023
There was a problem hiding this comment.
@danny0405 Please review again, i've moved the check based on your suggestion.
| @@ -1221,35 +1222,6 @@ public void testArchiveTableWithMetadataTableCompaction() throws Exception { | |||
| commitsAfterArchival); | |||
| } | |||
|
|
|||
| @Test | |||
| public void testArchiveCommitsWithCompactionCommitInMetadataTableTimeline() throws Exception { | |||
There was a problem hiding this comment.
this test is redundant and uses dummy instants. we're already checking archival with compaction in metadata table in several other tests such as testArchiveTableWithMetadataTableCompaction and testArchivalAndCompactionInMetadataTable
|
Thanks for the contribution @codope , I have reviewed and created a patch: |
danny0405
approved these changes
Nov 5, 2023
danny0405
reviewed
Nov 5, 2023
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
Outdated
Show resolved
Hide resolved
4 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Metadata table (MDT) has special handling for compaction. This PR ensures MDT compaction is handled in completion time based filesystem view. Previously, out-of-order commit tests were disabled. This PR now enables that test to show the file slicing happens correctly.
NOTE: This PR does not remove the limitation of single writer or OCC concurrency mode from MDT. That will be tackled separately.
Impact
Support out of order commits due to compaction in MDT with completion time view.
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist