[HUDI-5859] Adding standalone restore tool by nsivabalan · Pull Request #8044 · apache/hudi

nsivabalan · 2023-02-25T04:13:57Z

Change Logs

For MOR Table, restoring to a very old delta commit is very time consuming. since internally, we do rollback of 1 commit at a time. This standalone tool takes a stab at improving the performance of restore. You can choose a delta commit just before a compaction commit, and this tool will directly delete files for newer file slices after the delta commit chosen.

this tool does not yet suport restoring to middle of file slice.
Restore timestamp has to be latest delta commit before any compaction commit.

For eg,
dc1,
dc2
c3,
dc4,
dc5,
c6,
dc7,
dc8,
c9,
dc10,
dc11

Valid commit times to restore w/ this tool:
dc2, or dc5, dc8.

In other words, this tool can only clean up entire file slices and hence.

After cleaning up the data files, this toll will also delete the corresponding commit meta files from ".hoodie".

Caution:
Metadata has to be disbaled.
And this tool takesn unconventional route of not going via rollback. This tool directly lists the files and deletes them and also deleted the timeline files if necessary.

Sample command

./bin/spark-submit --num-executors 1 --executor-cores 2 --deploy-mode client --driver-memory 8g --executor-memory 8g  --class org.apache.hudi.utilities.MORRestoreTool BUNDLE_LOCATION/hudi-utilities-bundle_2.11-0.14.0-SNAPSHOT.jar --base-path /tmp/hudi_trips_more_restore/ --parallelism 4 --commitTime 20230225091008404 --spark-master local[2] --execute --cleanUpMetadata

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

nsivabalan · 2023-02-25T04:18:33Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/MORRestoreTool.java

+          break;
+        } else {
+          // we need to collect only partial list of log files to be deleted
+          /*TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);


It is feasible to support restoring to any delta commit. For now, have not tested this part yet.

the-other-tim-brown · 2023-02-25T17:45:11Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java

   * Clears hoodie.table.metadata.partitions in hoodie.properties
   */
-  private void clearMetadataTablePartitionsConfig(Option<MetadataPartitionType> partitionType, boolean clearAll) {
+  public static void clearMetadataTablePartitionsConfig(Option<MetadataPartitionType> partitionType, boolean clearAll, HoodieTableMetaClient metaClient) {


Let's avoid moving methods to static. This only makes the code harder to unit test. Also changing from private to public static seems like maybe we should move this functionality outside of the HoodieTable class, what do you think?

the-other-tim-brown · 2023-02-25T17:49:29Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/MORRestoreTool.java

+  private final HoodieTableMetaClient metaClient;
+
+  public MORRestoreTool(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;


This constructor isn't setting the other instance vars, can we set those to avoid NPEs? Are there some vars that only need to exist within the constructor?

the-other-tim-brown · 2023-02-25T17:59:34Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/TestMorRestoreTool.java

+        }
+      }
+    }
+  }


Can you add some assertions that the table can still be read and that the records retrieved match expectations?

Are there any assertions on the metadata table that also need to be added here to make sure that has the correct end result/state?

the-other-tim-brown · 2023-02-25T18:00:54Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/TestMorRestoreTool.java

+    DATA_GENERATOR.close();
+  }
+
+  @Test


Add a sanity check test that the "dry run" is taken into account

the-other-tim-brown · 2023-02-25T19:10:12Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/MORRestoreTool.java

+    HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+    FileSystem fs = metaClient.getFs();
+    if (fs.exists(new Path(basePath + "/" + METADATA_TABLE_FOLDER_PATH))) {
+      if (!cfg.cleanupMetadata) {


Should there be a dryrun option for this?

the-other-tim-brown · 2023-02-25T22:21:15Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/MORRestoreTool.java

+                filesToDelete.add(Pair.of(pPath, logFile.getPath().toString()));
+              });
+              LOG.info(fileSlice.getBaseInstantTime() + " Not processing remaining file slices");
+              break;


what is the purpose of using break here and below?

hudi-bot · 2023-02-28T08:28:23Z

CI report:

22c99b3 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

vinothchandar · 2023-03-01T02:13:07Z

Lets build this as a CLI enhancement? and also fr COW and MOR in general.

Adding standalone restore tool

353b4f3

nsivabalan commented Feb 25, 2023

View reviewed changes

Leveraging spark engine to parallelize

caba5a8

nsivabalan force-pushed the restoreTool branch 2 times, most recently from 038d577 to 8742a53 Compare February 25, 2023 17:39

Adding support to cleanup metadata table

1025352

nsivabalan force-pushed the restoreTool branch from 8742a53 to 1025352 Compare February 25, 2023 17:40

the-other-tim-brown reviewed Feb 25, 2023

View reviewed changes

adhoc fixes

624c41c

the-other-tim-brown reviewed Feb 25, 2023

View reviewed changes

Fixing logging and adding savepoint validation

22c99b3

nsivabalan changed the title ~~[WIP] Adding standalone restore tool~~ [HUDI-5859] Adding standalone restore tool Feb 28, 2023

xushiyan self-assigned this Feb 28, 2023

nsivabalan closed this Mar 16, 2023

hudi-bot mentioned this pull request Dec 9, 2025

Faster restore tool #15800

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[HUDI-5859] Adding standalone restore tool#8044

[HUDI-5859] Adding standalone restore tool#8044
nsivabalan wants to merge 5 commits intoapache:masterfrom
nsivabalan:restoreTool

nsivabalan commented Feb 25, 2023 •

edited

Loading

Uh oh!

nsivabalan Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

the-other-tim-brown Feb 25, 2023

Uh oh!

hudi-bot commented Feb 28, 2023

Uh oh!

vinothchandar commented Mar 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

Conversation

nsivabalan commented Feb 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Feb 28, 2023

CI report:

Uh oh!

vinothchandar commented Mar 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nsivabalan commented Feb 25, 2023 •

edited

Loading