Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enrich-kinesis: improve sink retry policy for when throttled by kinesis #697

Closed
istreeter opened this issue Sep 30, 2022 · 0 comments
Closed

Comments

@istreeter
Copy link
Contributor

We had a pipeline which was processing a large number of events, but was configured with only one output kinesis shard. The events were also fairly large in bytes. We noticed a couple of things:

  1. The app crashed repeatedly, because it got throttled by kinesis and then exceeded the number of retries for writing to the stream.
  2. Based on cloudwatch metrics, we never seemed to hit the maximum allowed write capacity of the shard.

On point 1, we could simply bump the default number of retries. But I think we should do more than this.... I propose we make a custom retry policy, so if the exception is a ProvisionedThroughputExceededException then we should never let the app crash, but instead keep on retrying. The configuration parameter maxRetries should only apply to other unexpected exceptions.

On point 2, this might be misleading, because if the app was crashing in less than 1 minute, then our minutely rate of writing to kinesis would appear to be low. We should check that after fixing the retry policy then we max out the write capacity of the shard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant