[HUDI-5477] Optimize timeline loading in Hudi sync client #7561
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yihua
added
meta-sync
priority:critical
yihua
changed the title
danny0405
reviewed
Dec 28, 2022
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
Show resolved
Hide resolved
danny0405
reviewed
Dec 28, 2022
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
Outdated
Show resolved
Hide resolved
yihua
changed the title
yihua
force-pushed
the
HUDI-5477-optimize-timeline-loading-in-sync
branch
from
40361ca
to
f61d948
Compare
danny0405
approved these changes
Jan 4, 2023
|
CI passes. Merging this PR |
XuQianJin-Stars
pushed a commit
that referenced
this pull request
Jan 31, 2023
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in .hoodie/archived folder. This change improves the timeline loading by (1) only reading active timeline if the last sync time is the same as or after the start of the active timeline; (2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline. (cherry picked from commit ab61f61)
This was referenced Mar 3, 2023
nsivabalan
pushed a commit
to nsivabalan/hudi
that referenced
this pull request
Mar 22, 2023
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in .hoodie/archived folder. This change improves the timeline loading by (1) only reading active timeline if the last sync time is the same as or after the start of the active timeline; (2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline.
fengjian428
pushed a commit
to fengjian428/hudi
that referenced
this pull request
Apr 5, 2023
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in .hoodie/archived folder. This change improves the timeline loading by (1) only reading active timeline if the last sync time is the same as or after the start of the active timeline; (2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline.
BruceKellan
reviewed
May 24, 2023
| HoodieTableMetaClient metaClient, String exclusiveStartInstantTime) { | ||
| HoodieActiveTimeline activeTimeline = metaClient.getActiveTimeline(); | ||
| HoodieDefaultTimeline timeline = | ||
| activeTimeline.isBeforeTimelineStarts(exclusiveStartInstantTime) |
There was a problem hiding this comment.
@yihua I have a doubt, since rollback and commit are archived separately, is it possible that there is a very early rollback instant, causing activeTimeline.isBeforeTimelineStarts(exclusiveStartInstantTime) to return false?
There was a problem hiding this comment.
@BruceKellan You bring up a good point. So you are talking about the following case where the timeline is like:
ts3.rollback, ts50.commit, ts51.commit, ts52.commit, ...
and ts49.commit and ts48.commit are archived.
If we pass in ts47 as the exclusiveStartInstantTime at this point, activeTimeline.isBeforeTimelineStarts(exclusiveStartInstantTime) returns false. However, the active timeline misses ts48.commit and ts49.commit which are required for meta sync. This is a problem if ts48.commit or ts49.commit has partition changes.
I'll put up a fix on that. Thanks for catching it!
There was a problem hiding this comment.
Here's the fix: #8991
4 tasks
yihua
added a commit
that referenced
this pull request
Jun 16, 2023
yihua
added a commit
to yihua/hudi
that referenced
this pull request
Aug 24, 2023
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in .hoodie/archived folder. This change improves the timeline loading by (1) only reading active timeline if the last sync time is the same as or after the start of the active timeline; (2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline.
yihua
added a commit
to yihua/hudi
that referenced
this pull request
Aug 24, 2023
…a sync (apache#8991) This commit fixes the problematic API implementation (TimelineUtils.getCommitsTimelineAfter) introduced by apache#7561.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in
.hoodie/archivedfolder. Such behavior is introduced by #6662.This PR improves the timeline loading by
(1) only reading active timeline if the last sync time is the same as or after the start of the active timeline;
(2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline.
Impact
This PR improves the performance of metastore sync.
Risk level
low
Documentation Update
N/A
Contributor's checklist