Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KafkaConsumer: triggerCount should specify the exact value for initiating a checkpoint #45

Closed
cancilla opened this issue Sep 18, 2017 · 2 comments
Assignees

Comments

@cancilla
Copy link
Collaborator

In the current implementation of the triggerCount parameter, the value specified by the user will be the minimum number of tuples that will be submitted prior to initiating a checkpoint. This implies that the actual number of tuples that get submitted before the operator is checkpointed is not deterministic.

The KafkaConsumer operator needs to be enhanced so that the value specified for triggerCount represents the exact number of tuples that will be submitted prior to a checkpoint.

@cancilla
Copy link
Collaborator Author

cancilla commented Oct 6, 2017

Fixed in v1.2.0

@cancilla cancilla closed this as completed Oct 6, 2017
@ddebrunner
Copy link
Member

What's the rationale for requiring the number be fixed?

I'm looking at if this change broke the operator at least once guarantee in a consistent region.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants