[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #7997

voonhous · 2023-02-20T11:47:04Z

...ndle

Applying the fix from #5185 will fix write issues for MOR tables, but will cause write issues for COW tables.

More information on how to reproduce the COW error in this jira issue:
https://issues.apache.org/jira/browse/HUDI-5822

This addresses the issue raised here: #5782

Change Logs

Complement the change of getLatestFileSlices -> getAllFileGroups in [HUDI-3758] Fix duplicate fileId error in MOR table type with flink bucket hash Index #5185

Impact

Allow jobs to recover properly
Ensure that the correct fileSlices are being read

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java

...tasource/hudi-flink/src/main/java/org/apache/hudi/sink/bucket/BucketStreamWriteFunction.java

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java

SteNicholas · 2023-02-22T08:32:23Z

@voonhous, could you please add the test cases for the changes?

voonhous · 2023-02-22T08:33:53Z

@voonhous, could you please add the test cases for the changes?

Hmmm, not easy to write a test for this. This bug can only be triggered if JM's rollback is running behind TM.

voonhous · 2023-02-22T10:31:14Z

@hbgstc123 @TengHuo for visibility

SteNicholas · 2023-02-22T13:16:04Z

@voonhous, could you add the unit tests for bootstrapIndexIfNeed?

voonhous · 2023-02-24T10:39:34Z

@SteNicholas @danny0405 Can you please help to review this commit? Thank you

voonhous · 2023-02-24T14:46:56Z

@hudi-bot run azure

hudi-bot · 2023-02-24T17:28:54Z

CI report:

4ea6533 Azure: FAILURE Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java

voonhous · 2023-02-28T09:57:53Z

...-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java

@@ -115,16 +115,24 @@ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePat
    // TODO(HUDI-1517) use provided marker-file's path instead
    Option<HoodieLogFile> latestLogFileOption = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
        HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime);
+
+    // Log file can be deleted if the commit to rollback is also the commit that created the fileGroup


@danny0405 here

Okay, seems here is the code you wanna folk from the listing based rollback strategy:

// In case all data was inserts and the commit failed, delete the file belonging to that commit // We do not know fileIds for inserts (first inserts are either log files or base files), // delete all files for the corresponding failed commit, if present (same as COW) hoodieRollbackRequests.add(getHoodieRollbackRequest(partitionPath, filesToDelete));

Can we split this PR into 2, one is for the marker based rollback fix, another is for the testing of flink stream writer with hashing index enabled? I mean we create two JIRA issues here.

Sure, good idea!

separate PR: #8077

danny0405 · 2023-03-20T06:34:21Z

Close because it should be fixed in #8077

voonhous · 2023-03-20T07:24:22Z

@danny0405 This is not fixed... The rollback standardises the rollback logic between markers and listing for MOR tables.

The bug is still present in COW tables... TM might run ahead of of JM. If we continue to use getAllFileGroups, TM might still be able to fetch the fileGroup that is destined to be deleted, causing the fileId/Group not found in issue.

danny0405 · 2023-03-20T07:56:13Z

TM might run ahead of of JM

How could this happen? The write tasks on JM would wait for the JM coordinator to finish initialization then start to hande the writiing process.

voonhous · 2023-03-20T08:23:35Z

JM

2023-02-20 11:36:34,826 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Begin rollback of instant 20230220112929727
2023-02-20 11:36:34,833 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hudi_path
2023-02-20 11:36:34,947 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://hudi_path/.hoodie/hoodie.properties
2023-02-20 11:36:34,952 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hudi_path
2023-02-20 11:36:34,952 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hudi_path
2023-02-20 11:36:35,320 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:35,321 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:35,321 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:35,323 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling Rollback at instant time : 20230220113634829 (exists in active timeline: true), with rollback plan: false
2023-02-20 11:36:35,612 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220113634829__rollback__REQUESTED]}
2023-02-20 11:36:35,612 INFO  org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor [] - Requesting Rollback with instant time [==>20230220113634829__rollback__REQUESTED]
2023-02-20 11:36:35,620 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220113634829__rollback__REQUESTED]}
2023-02-20 11:36:35,694 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.requested
2023-02-20 11:36:35,706 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.inflight
2023-02-20 11:36:35,709 INFO  org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor [] - Clean out all base files generated for commit: [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,720 INFO  org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor [] - Time(in ms) taken to finish rollback 11
2023-02-20 11:36:35,720 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Rolled back inflight instant 20230220112929727
2023-02-20 11:36:35,721 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Index rolled back for commits [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,725 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Deleting instant=[==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,725 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Deleting instant [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,728 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Removed instant [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,728 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Deleting instant [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,731 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Removed instant [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,732 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Deleted pending commit [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,733 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.inflight
2023-02-20 11:36:35,786 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?hdfs://hudi_path/.hoodie/20230220113634829.rollback
2023-02-20 11:36:35,786 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Rollback of Commits [20230220112929727] is complete
2023-02-20 11:36:35,805 INFO  org.apache.hudi.common.fs.FSUtils                            [] - Removed directory at hdfs://hudi_path/.hoodie/.temp/20230220112929727
2023-02-20 11:36:35,806 INFO  org.apache.hudi.metrics.HoodieMetrics                        [] - Sending rollback metrics (duration=973, numFilesDeleted=2)
2023-02-20 11:36:35,812 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Generate a new instant time: 20230220113635812 action: commit
2023-02-20 11:36:35,815 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230220113634829__rollback__COMPLETED]}

TM

 11:36:33,837 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hdfs_path
2023-02-20 11:36:33,837 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hdfs_path
2023-02-20 11:36:33,840 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,841 INFO  org.apache.hudi.client.BaseHoodieClient                      [] - Embedded Timeline Server is disabled. Not starting timeline service
2023-02-20 11:36:33,843 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:33,843 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:33,849 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Took 2 ms to read  0 instants, 0 replaced file groups
2023-02-20 11:36:33,850 INFO  org.apache.hudi.sink.common.AbstractStreamWriteFunction      [] - Send bootstrap write metadata event to coordinator, task[0].
2023-02-20 11:36:33,850 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,853 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - bucket_write: dim_buyer_info_test (1/2)#0 (e2e7a38d69393b8d814ad48544573435_829468138015e9cb689e833f1800885e_0_0) switched from INITIALIZING to RUNNING.
2023-02-20 11:36:33,857 INFO  org.apache.hudi.sink.CleanFunction                           [] - Executor executes action [wait for cleaning finish] success!
2023-02-20 11:36:33,860 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Loading Hoodie Table dim_buyer_info_test, with path hdfs://hdfs_path/age=0
2023-02-20 11:36:33,860 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hdfs_path
2023-02-20 11:36:33,867 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://hdfs_path/.hoodie/hoodie.properties
2023-02-20 11:36:33,868 INFO  org.apache.hudi.common.util.ClusteringUtils                  [] - Found 0 files in pending clustering operations
2023-02-20 11:36:33,868 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Building file system view for partition (age=1)
2023-02-20 11:36:33,872 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hdfs_path
2023-02-20 11:36:33,872 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hdfs_path
2023-02-20 11:36:33,878 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,879 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - addFilesToView: NumFiles=2, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=1
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - bootstrapIndexIfNeed with timeline: [[==>20230220112929727__commit__INFLIGHT]]
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hdfs_path
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Should load this partition bucket 0 with fileID 00000000-ee86-4b41-a704-9e075dd253d8
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Adding fileID 00000000-ee86-4b41-a704-9e075dd253d8 to the bucket 0 of partition age=1.

Explanation

TM's BucketStreamWriteFunction#processElement is executing at 2023-02-20 11:36:33,880, while JM completed the rollback at 2023-02-20 11:36:35,815 (2 seconds after TM's BucketStreamWriteFunction#processElement)

Hope this log paints a clearer picture.

voonhous · 2023-03-21T05:01:00Z

@danny0405 Running the tests included with this PR without the fix fails... Hence, I believe the fix in this PR is still required.

danny0405 · 2023-03-21T05:04:10Z

@danny0405 Running the tests included with this PR without the fix fails... i.e. The fix in this PR is still required.

Got ya, can you rebase with the latest master and force-push again

…oodieMergeHandle

voonhous · 2023-03-21T05:54:33Z

@danny0405 Running the tests included with this PR without the fix fails... i.e. The fix in this PR is still required.

Got ya, can you rebase with the latest master and force-push again

Rebased and force-pushed.

danny0405 · 2023-03-21T06:59:35Z

@hudi-bot run azure

danny0405 · 2023-03-21T09:03:34Z

@hudi-bot run azure

danny0405 · 2023-03-21T12:40:15Z

@voonhous The CI can not be triggered, can you fire another PR instead.

…oodieMergeHandle

voonhous · 2023-03-22T02:52:17Z

@voonhous The CI can not be triggered, can you fire another PR instead.

#8263

SteNicholas reviewed Feb 20, 2023

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java Outdated Show resolved Hide resolved

SteNicholas reviewed Feb 20, 2023

View reviewed changes

...tasource/hudi-flink/src/main/java/org/apache/hudi/sink/bucket/BucketStreamWriteFunction.java Show resolved Hide resolved

voonhous marked this pull request as draft February 20, 2023 13:00

voonhous marked this pull request as ready for review February 20, 2023 13:54

voonhous changed the title ~~Fix FileId not found exception when FileId is passed to HoodieMergeHa…~~ [HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa… Feb 20, 2023

voonhous marked this pull request as draft February 20, 2023 13:56

voonhous force-pushed the HUDI-5822 branch from 907d6ed to 4ea6533 Compare February 21, 2023 10:39

voonhous changed the title ~~[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa…~~ [HUDI-5822] Fix write and read correctness issue when a rollback is performed Feb 21, 2023

bvaradar added the pr:wip Work in Progress/PRs label Feb 22, 2023

bvaradar self-assigned this Feb 22, 2023

bvaradar requested changes Feb 22, 2023

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java Outdated Show resolved Hide resolved

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java Outdated Show resolved Hide resolved

voonhous force-pushed the HUDI-5822 branch from df9cfdb to f4e65e6 Compare February 22, 2023 07:38

voonhous changed the title ~~[HUDI-5822] Fix write and read correctness issue when a rollback is performed~~ [HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa Feb 22, 2023

voonhous changed the title ~~[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa~~ [HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... Feb 22, 2023

voonhous marked this pull request as ready for review February 22, 2023 07:50

voonhous requested review from bvaradar and SteNicholas and removed request for bvaradar and SteNicholas February 22, 2023 08:04

voonhous force-pushed the HUDI-5822 branch from da7f7d2 to b70a1a0 Compare February 24, 2023 18:24

danny0405 reviewed Feb 25, 2023

View reviewed changes

hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java Outdated Show resolved Hide resolved

voonhous commented Feb 28, 2023

View reviewed changes

voonhous mentioned this pull request Mar 1, 2023

[HUDI-5862] Strength MARKER based rollback for log files with a single commit #8077

Merged

4 tasks

danny0405 closed this Mar 20, 2023

danny0405 reopened this Mar 21, 2023

[HUDI-5822] Fix FileId not found exception when FileId is passed to H…

285de8f

…oodieMergeHandle

voonhous force-pushed the HUDI-5822 branch from b64c123 to 285de8f Compare March 21, 2023 05:53

[HUDI-5822] Fix FileId not found exception when FileId is passed to H…

6aa7bc9

…oodieMergeHandle

voonhous marked this pull request as draft March 22, 2023 02:47

voonhous marked this pull request as ready for review March 22, 2023 02:48

voonhous closed this Mar 22, 2023

voonhous mentioned this pull request Mar 22, 2023

[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #8263

Merged

4 tasks

voonhous deleted the HUDI-5822 branch April 28, 2023 07:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #7997

[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #7997

voonhous commented Feb 20, 2023 •

edited

Loading

SteNicholas commented Feb 22, 2023

voonhous commented Feb 22, 2023

voonhous commented Feb 22, 2023

SteNicholas commented Feb 22, 2023

voonhous commented Feb 24, 2023

voonhous commented Feb 24, 2023

hudi-bot commented Feb 24, 2023

voonhous Feb 28, 2023

danny0405 Feb 28, 2023 •

edited

Loading

voonhous Feb 28, 2023

danny0405 Feb 28, 2023 •

edited

Loading

voonhous Feb 28, 2023

voonhous Mar 1, 2023

danny0405 commented Mar 20, 2023

voonhous commented Mar 20, 2023

danny0405 commented Mar 20, 2023

voonhous commented Mar 20, 2023 •

edited

Loading

voonhous commented Mar 21, 2023 •

edited

Loading

danny0405 commented Mar 21, 2023

voonhous commented Mar 21, 2023

danny0405 commented Mar 21, 2023

danny0405 commented Mar 21, 2023

danny0405 commented Mar 21, 2023

voonhous commented Mar 22, 2023

[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #7997

[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa... #7997

Conversation

voonhous commented Feb 20, 2023 • edited Loading

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

SteNicholas commented Feb 22, 2023

voonhous commented Feb 22, 2023

voonhous commented Feb 22, 2023

SteNicholas commented Feb 22, 2023

voonhous commented Feb 24, 2023

voonhous commented Feb 24, 2023

hudi-bot commented Feb 24, 2023

CI report:

voonhous Feb 28, 2023

Choose a reason for hiding this comment

danny0405 Feb 28, 2023 • edited Loading

Choose a reason for hiding this comment

voonhous Feb 28, 2023

Choose a reason for hiding this comment

danny0405 Feb 28, 2023 • edited Loading

Choose a reason for hiding this comment

voonhous Feb 28, 2023

Choose a reason for hiding this comment

voonhous Mar 1, 2023

Choose a reason for hiding this comment

danny0405 commented Mar 20, 2023

voonhous commented Mar 20, 2023

danny0405 commented Mar 20, 2023

voonhous commented Mar 20, 2023 • edited Loading

JM

TM

Explanation

voonhous commented Mar 21, 2023 • edited Loading

danny0405 commented Mar 21, 2023

voonhous commented Mar 21, 2023

danny0405 commented Mar 21, 2023

danny0405 commented Mar 21, 2023

danny0405 commented Mar 21, 2023

voonhous commented Mar 22, 2023

voonhous commented Feb 20, 2023 •

edited

Loading

danny0405 Feb 28, 2023 •

edited

Loading

danny0405 Feb 28, 2023 •

edited

Loading

voonhous commented Mar 20, 2023 •

edited

Loading

voonhous commented Mar 21, 2023 •

edited

Loading