[HUDI-7235] Fix checkpoint bug for S3/GCS Incremental Source by vinishjail97 · Pull Request #10336 · apache/hudi

vinishjail97 · 2023-12-15T16:38:40Z

Change Logs

Fix bug in checkpointing logic for S3/GCS in empty dataset use-case. The reason for the bug was following.

1st delta commit's checkpoint, processed 3 files.

23/12/06 16:55:26 INFO S3EventsHoodieIncrSource  : Querying S3 with:00000000000000, queryInfo:Query information for Incremental Source queryType: snapshot, previousInstant: 00000000000000, startInstant: 00000000000000, endInstant: 20231206150423946, orderColumn: _hoodie_commit_time, keyColumn: s3.object.key, limitColumn: s3.object.size, orderByColumns: [_hoodie_commit_time, s3.object.key]
	

23/12/06 16:55:42 INFO S3EventsHoodieIncrSource  : Adjusting end checkpoint:20231206150423946 based on sourceLimit :300000000
	
23/12/06 16:55:46 INFO S3EventsHoodieIncrSource  : Adjusted end checkpoint :20231206150423946#ee-facts/0012_part_00.parquet
	

23/12/06 16:55:49 INFO S3EventsHoodieIncrSource  : Total number of files to process :3

2nd delta commit was an empty one and the checkpoint returned was 20231206150423946 which is not a valid checkpoint progression because it should either be equal or increase monotonically (based on lexicographical order)

23/12/06 16:59:52 INFO S3EventsHoodieIncrSource  : Querying S3 with:20231206150423946#ee-facts/0012_part_00.parquet, queryInfo:Query information for Incremental Source queryType: incremental, previousInstant: 00000000000000, startInstant: 20231206150423946, endInstant: 20231206150423946, orderColumn: _hoodie_commit_time, keyColumn: s3.object.key, limitColumn: s3.object.size, orderByColumns: [_hoodie_commit_time, s3.object.key]
	
23/12/06 16:59:53 INFO S3EventsHoodieIncrSource  : Adjusting end checkpoint:20231206150423946 based on sourceLimit :300000000
	
23/12/06 16:59:55 INFO S3EventsHoodieIncrSource  : Empty source, returning endpoint:20231206150423946

As the previous commits' checkpoint was a faulty one, the 3rd commit read the same set of files again and wrote duplicate data.

Impact

Bug Fix

Risk level (write none, low medium or high below)

Medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

None, this is a bug fix for an existing feature.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

bvaradar

@vinishjail97 : Very good catch. Few questions and nit comments. Please take a look at the failing tests including the validate-PR

bvaradar · 2023-12-15T19:24:54Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/GcsEventsHoodieIncrSource.java

    if (!checkPointAndDataset.getRight().isPresent()) {
-      LOG.info("Empty source, returning endpoint:" + queryInfo.getEndInstant());
-      return Pair.of(Option.empty(), queryInfo.getEndInstant());
+      LOG.info("Empty source, returning endpoint:" + checkPointAndDataset.getLeft().toString());


nit: .toString() is not needed in log message

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsHoodieIncrSource.java

bvaradar · 2023-12-15T19:26:14Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java

    if (sourceData.isEmpty()) {
-      return Pair.of(cloudObjectIncrCheckpoint, Option.empty());
+      // There is no file matching the prefix.
+      return Pair.of(new CloudObjectIncrCheckpoint(queryInfo.getEndInstant(), null), Option.empty());


Why can't we use the cloudObjectIncrCheckpoint that is being passed to this function ?

Fixed the code to pick the right checkpoint

bvaradar · 2024-04-01T18:41:04Z

@vinishjail97 : Can you address these comments and land it.

bvaradar · 2024-04-03T06:14:44Z

@vinishjail97 : Can you address these comments and land it.

Updated the diff after addressing review comments.

bvaradar

I have fixed the issue in additional place when there is no file matching the prefix.

hudi-bot · 2024-04-23T06:35:37Z

CI report:

de49a9d UNKNOWN
1b754df Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

Co-authored-by: Balaji Varadarajan <balaji.varadarajan@robinhood.com> Co-authored-by: Balaji Varadarajan <vbalaji@apache.org>

bvaradar self-assigned this Dec 15, 2023

bvaradar requested changes Dec 15, 2023

View reviewed changes

github-actions bot added the size:S PR with lines of changes in (10, 100] label Feb 26, 2024

bvaradar force-pushed the HUDI-7235-Fix-Checkpoint-Bug branch from 7f685a6 to de49a9d Compare April 3, 2024 06:10

bvaradar force-pushed the HUDI-7235-Fix-Checkpoint-Bug branch from 0656fa6 to 1bb5e22 Compare April 5, 2024 20:48

vinishjail97 and others added 6 commits April 16, 2024 21:31

Fix checkpoint bug for S3/GCS Incremental Source

1a966a5

Fix tests

7e51d73

Address comments

4196612

Fix check-style bug

8d4d5b1

Set correct checkpoint even when sourceData is empty

3e72f48

Set correct checkpoint even when sourceData is empty

5edeb0f

bvaradar force-pushed the HUDI-7235-Fix-Checkpoint-Bug branch from 1bb5e22 to 5edeb0f Compare April 17, 2024 04:32

Balaji Varadarajan added 2 commits April 16, 2024 23:43

Fix test failure

47b6803

Fix unit-test failure

1b754df

bvaradar approved these changes Apr 23, 2024

View reviewed changes

bvaradar merged commit fb02249 into apache:master Apr 25, 2024

yihua pushed a commit that referenced this pull request May 15, 2024

[HUDI-7235] Fix checkpoint bug for S3/GCS Incremental Source (#10336)

6a84e7c

Co-authored-by: Balaji Varadarajan <balaji.varadarajan@robinhood.com> Co-authored-by: Balaji Varadarajan <vbalaji@apache.org>

yihua pushed a commit that referenced this pull request May 15, 2024

[HUDI-7235] Fix checkpoint bug for S3/GCS Incremental Source (#10336)

d42f399

Co-authored-by: Balaji Varadarajan <balaji.varadarajan@robinhood.com> Co-authored-by: Balaji Varadarajan <vbalaji@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-7235] Fix checkpoint bug for S3/GCS Incremental Source#10336

[HUDI-7235] Fix checkpoint bug for S3/GCS Incremental Source#10336
bvaradar merged 8 commits intoapache:masterfrom
vinishjail97:HUDI-7235-Fix-Checkpoint-Bug

vinishjail97 commented Dec 15, 2023 •

edited by bvaradar

Loading

Uh oh!

bvaradar left a comment

Uh oh!

bvaradar Dec 15, 2023

Uh oh!

Uh oh!

bvaradar Dec 15, 2023

Uh oh!

bvaradar Apr 3, 2024 •

edited

Loading

Uh oh!

bvaradar commented Apr 1, 2024

Uh oh!

bvaradar commented Apr 3, 2024

Uh oh!

bvaradar left a comment

Uh oh!

hudi-bot commented Apr 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vinishjail97 commented Dec 15, 2023 • edited by bvaradar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

bvaradar left a comment

Choose a reason for hiding this comment

Uh oh!

bvaradar Dec 15, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bvaradar Dec 15, 2023

Choose a reason for hiding this comment

Uh oh!

bvaradar Apr 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bvaradar commented Apr 1, 2024

Uh oh!

bvaradar commented Apr 3, 2024

Uh oh!

bvaradar left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Apr 23, 2024

CI report:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vinishjail97 commented Dec 15, 2023 •

edited by bvaradar

Loading

bvaradar Apr 3, 2024 •

edited

Loading