You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We had a pipeline which was processing a large number of events, but was configured with only one output kinesis shard. The events were also fairly large in bytes. We noticed a couple of things:
The app crashed repeatedly, because it got throttled by kinesis and then exceeded the number of retries for writing to the stream.
Based on cloudwatch metrics, we never seemed to hit the maximum allowed write capacity of the shard.
On point 1, we could simply bump the default number of retries. But I think we should do more than this.... I propose we make a custom retry policy, so if the exception is a ProvisionedThroughputExceededException then we should never let the app crash, but instead keep on retrying. The configuration parameter maxRetries should only apply to other unexpected exceptions.
On point 2, this might be misleading, because if the app was crashing in less than 1 minute, then our minutely rate of writing to kinesis would appear to be low. We should check that after fixing the retry policy then we max out the write capacity of the shard.
The text was updated successfully, but these errors were encountered:
We had a pipeline which was processing a large number of events, but was configured with only one output kinesis shard. The events were also fairly large in bytes. We noticed a couple of things:
On point 1, we could simply bump the default number of retries. But I think we should do more than this.... I propose we make a custom retry policy, so if the exception is a
ProvisionedThroughputExceededException
then we should never let the app crash, but instead keep on retrying. The configuration parametermaxRetries
should only apply to other unexpected exceptions.On point 2, this might be misleading, because if the app was crashing in less than 1 minute, then our minutely rate of writing to kinesis would appear to be low. We should check that after fixing the retry policy then we max out the write capacity of the shard.
The text was updated successfully, but these errors were encountered: