Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-7618] Add ability to ignore checkpoints in delta streamer #11018

Merged
merged 7 commits into from
Apr 19, 2024
Merged

[HUDI-7618] Add ability to ignore checkpoints in delta streamer #11018

merged 7 commits into from
Apr 19, 2024

Conversation

sampan-s-nayak
Copy link
Contributor

Change Logs

  • Added new param --ignore-checkpoint in hoodieStreamer and logic to ignore checkpoint when param is set
  • Added a couple of tests for the new changes

Impact

no major impact as this is a new feature

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:M PR with lines of changes in (100, 300] label Apr 15, 2024
nsivabalan
nsivabalan approved these changes Apr 16, 2024
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 minor nit

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamer.java Outdated Show resolved Hide resolved
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamer.java Outdated Show resolved Hide resolved
@github-actions github-actions bot added size:S PR with lines of changes in (10, 100] and removed size:M PR with lines of changes in (100, 300] labels Apr 18, 2024
@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@nsivabalan nsivabalan merged commit ca77fda into apache:master Apr 19, 2024
40 checks passed
@danny0405
Copy link
Contributor

danny0405 commented Apr 19, 2024 •
edited

Hi, @nsivabalan the master compile got crush with this patch, can you take care of it?

yihua pushed a commit that referenced this pull request May 15, 2024
@sampan-s-nayak @yihua
[HUDI-7618] Add ability to ignore checkpoints in delta streamer (#11018)
5582cde
yihua pushed a commit that referenced this pull request May 15, 2024
@sampan-s-nayak @yihua
[HUDI-7618] Add ability to ignore checkpoints in delta streamer (#11018)
82c3209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nsivabalan nsivabalan nsivabalan approved these changes

@rmahindra123 rmahindra123 rmahindra123 approved these changes

Assignees
No one assigned
Labels
release-0.15.0 size:S PR with lines of changes in (10, 100]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@sampan-s-nayak @hudi-bot @danny0405 @nsivabalan @rmahindra123