Exponential backoff retries for ProvisionedThroughputExceeded errors #222
Conversation
…putExceeded error
| elif response.status_code < 500: | ||
| # We don't retry on a ConditionalCheckFailedException or other 4xx because we assume they will | ||
| # fail in perpetuity. Retrying when there is already contention could cause other problems | ||
| elif response.status_code < 500 and code != 'ProvisionedThroughputExceededException': |
There was a problem hiding this comment.
let's make it configurable
|
@JohnEmhoff Are you still working on this? |
|
I haven't had a chance to update it to be configurable, no. Honestly I'm
not sure what the use case is to have it off. For example, when writing,
pynamo clears the batch object ahead of the write -- the application code
thus can't retry the writes that failed because it doesn't know what they
are.
…On Mar 1, 2017 3:52 PM, "Brandon Davidson" ***@***.***> wrote:
@JohnEmhoff <https://github.com/JohnEmhoff> Are you still working on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#222 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABnTd-BpiFtiHDljLUn5bJRIx0-5qYd1ks5rhdqkgaJpZM4LnKNL>
.
|
|
I"m doing this (some ugly variant anyway) in all my clients already, so definitely a +1 to merge. |
|
+1 here as well |
|
Any updates on this? Merging this addresses a known issue, which as of right now is a PITA to deal with. |
|
at Lyft we prefer backpressure to go to the client. i will merge since we actually don't use the retry within PynamoDB in most cases. it actually seems like a bigger bug that batch_write will clear on error. retry won't guarantee success, particularly when experiencing ProvisionedThroughputExceptions. can someone create an issue to track that in more detail? |
|
Agree that the batch_write clear on error is a more significant issue. However, this addresses a specific case we see regularly, which is that capacity provisioning lags the dynamodb I/O load and retries early on in the capacity scale-up phase eliminate a bunch of client-side retries. thanks! |
|
@JohnEmhoff @bedge @lukedeo @brandond released as part of 2.1.5 |
Currently this is a hard failure -- retry with backoff instead. Handling this client side is hard because
batch_write.commit()clears its list of pending operations on failure.Related issue: #218