[HUDI-7665] Support rolling upgrade to table version 8 by codope · Pull Request #12250 · apache/hudi

codope · 2024-11-13T18:37:44Z

Change Logs

Migrating table properties including partition fields, key generators, payload type, bootstrap index type. Handling both upgrade and downgrade
Migrating timeline to new layout: a) archived to LSM timeline layout, b) read both json/avro commit metadata, c) rename instants (including clustering action). These are all done for upgrade. For downgrade, I need to write a LSM to legacy archive timeline v1 writer.
Full compact the table, to get rid of log files. Both in case of upgrade and downgrade.
Drop version 7.
Some tests for above.

TODO:

LSM to legacy archived timeline writer to use in downgrade.
Migration path for CDC and incremental queries.
Handle differences between 0.x and 1.x for stuff needed in upgrade e.g compaction (need to compact older file slice), rollback (marker differences b/w 0.14 and 0.15). Though if we compact and delete any leftover markers, it might be okay. Need to test these scenarios.

Impact

Support rolling upgrade to table version 8.

Risk level (write none, low medium or high below)

high

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

danny0405 · 2024-11-14T08:11:46Z

...hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToEightUpgradeHandler.java

+                                             String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+    HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+    HoodieTableMetaClient metaClient = table.getMetaClient();
+    HoodieTableConfig tableConfig = metaClient.getTableConfig();


We can trigger a rollback for pending instants first, then remove the any log file markers explicitly if there are any.
This way we can clean up the table as much as possible before any other steps.

Ack, changed the code to rollback and compact in one step.

...hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToEightUpgradeHandler.java

danny0405 · 2024-11-14T08:15:12Z

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

+        throw new HoodieException(e);
+      }
+    };
+    lsmTimelineWriter.write(Collections.singletonList(ActiveAction.fromInstants(archivedTimeline.getInstants())),


The list for archived timeline is huge, we need to split the list into small batchs and write into the LSM timeline per-batch, by default, just use 10 instants as a batch which is in line with the current behavior.

Another concern is that the legacy archived timeline may contain enormous instants there (like several GBs of avro logs), it would be very time-consuming to load the whole legacy archived timeline, maybe we just load the latest avro log(which should be enough for file slicing).

Yes, i thought about it and I agree. I wanted to quickly do some local testing to validate the upgrade path. I will do the batching soon.

danny0405 · 2024-11-14T08:16:12Z

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

+
+    // Migrate the LSM timeline back to the old archived timeline format
+    try {
+      // TODO: Convert instants from the LSM format to the old format


We need timeline archiver v1 from Balaj's PR. cc @bvaradar

Ack. For now, i've ported over some legacy archiver code (copied from #11923) for local testing. Once that PR lands, I will rebase.

danny0405 · 2024-11-17T02:12:57Z

...t-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/TimelineArchiverV2.java

+  }
+
+  @Override
+  public int archiveInstants(HoodieEngineContext context, List<HoodieInstant> instantsToArchive, boolean acquireLock) throws IOException {


It looks like this acquireLock is always false.

it is true in UpgradeDowngradeUtils methods upgradeToLSMTimeline and downgradeFromLSMTimeline

why we need lock for upgrade/downgrade, is there any concurrency here?

port some legacy timeline archiver code and compact post rollback

hudi-bot · 2024-11-20T18:28:45Z

CI report:

4cfaa0a UNKNOWN
56bac12 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

codope · 2024-11-24T17:04:52Z

Closing in favor of #12327

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Nov 13, 2024

danny0405 reviewed Nov 14, 2024

View reviewed changes

...hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToEightUpgradeHandler.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 14, 2024

View reviewed changes

codope force-pushed the hudi-7665-table-props branch from af2c110 to 4cfaa0a Compare November 14, 2024 15:27

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Nov 14, 2024

codope force-pushed the hudi-7665-table-props branch 3 times, most recently from 12271aa to 9d9683f Compare November 16, 2024 14:05

danny0405 reviewed Nov 17, 2024

View reviewed changes

codope force-pushed the hudi-7665-table-props branch from 9d9683f to f5b1aa8 Compare November 17, 2024 15:31

codope added 2 commits November 20, 2024 22:38

[HUDI-7665] Support rolling upgrade to table version 8

c3fe3d4

port some legacy timeline archiver code and compact post rollback

Fix dowgrade logic and restore version 7

56bac12

codope force-pushed the hudi-7665-table-props branch from f5b1aa8 to 56bac12 Compare November 20, 2024 17:10

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels Nov 20, 2024

codope closed this Nov 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-7665] Support rolling upgrade to table version 8#12250

[HUDI-7665] Support rolling upgrade to table version 8#12250
codope wants to merge 2 commits intoapache:masterfrom
codope:hudi-7665-table-props

codope commented Nov 13, 2024 •

edited

Loading

Uh oh!

danny0405 Nov 14, 2024

Uh oh!

codope Nov 14, 2024

Uh oh!

Uh oh!

danny0405 Nov 14, 2024

Uh oh!

danny0405 Nov 14, 2024

Uh oh!

codope Nov 14, 2024

Uh oh!

danny0405 Nov 14, 2024

Uh oh!

codope Nov 14, 2024

Uh oh!

danny0405 Nov 17, 2024

Uh oh!

codope Nov 17, 2024

Uh oh!

danny0405 Nov 18, 2024

Uh oh!

hudi-bot commented Nov 20, 2024

Uh oh!

codope commented Nov 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

codope commented Nov 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Nov 20, 2024

CI report:

Uh oh!

codope commented Nov 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codope commented Nov 13, 2024 •

edited

Loading