-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-2888] Add the not-yet-fully-designed drain and checkpoint to runner comparison #8576
Conversation
SUCCESS --none-- |
Tagging many authors of other runners to review and potentially fill in details here, since I only know very high level bits. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kennknowles.
l1: | ||
l2: | ||
l3: | ||
- class: samza |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For Samza, it's using its native checkpoint capability. Basically checkpoint happens periodically based on user config. The checkpoints are saved in a metadata store or stream, and the states are flushed during checkpointing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can a user save it and then start up the pipeline later?
Ping. Any more to add? |
These features have been discussed a bit on the mailing list and I get asked about them a bit. I think it is OK to list them here like we list retractions / metadata triggers, both of which are further from reality than checkpoint and drain.
I would have preferred to do the whole comparison rewrite of https://issues.apache.org/jira/browse/BEAM-2888 because the tabular form misleadingly implies these are yes/no features, when in reality they have many facets. But I haven't had time, and haven't convinced anyone else to take on that work.
So this is a low-effort compromise.
What do you think?
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username
).[BEAM-XXX] Fixes bug in ApproximateQuantiles
, where you replaceBEAM-XXX
with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.