CU increases fail if decreases fail due to exceeded limits #103

Closed
fwachsmuth opened this Issue Jan 22, 2014 · 4 comments

Projects

None yet

2 participants

@fwachsmuth

When dynamic-dynamo intends to change read- and write-provisioning simultaneously where one increases and the other one decreases, the increase never happens due to exceeded decrease-limits.

In this case, the increase should still be executed as a "standalone" update call.

We just ran into massive throttlings in production due to this dead-lock situation.

@sebdah sebdah was assigned Jan 22, 2014
@sebdah sebdah added a commit that referenced this issue Jan 22, 2014
@sebdah Implemented a fix for tables #103 7c8990c
@sebdah sebdah added a commit that referenced this issue Jan 22, 2014
@sebdah Implemented #103 for GSI as well 5e8911a
@sebdah
Owner
sebdah commented Jan 22, 2014

The code has been cleaned and restructured. This functionality is now ported to support Global Secondary Indexes as well.

@fwachsmuth, can you please test with the branch called release/1.7.2 and report back when done? I'll make a proper release of this fix after that.

I've tested this (but with out any load, so no actual increase took place) and it looks good. Here's my log output.

2014-01-22 19:49:34,253 - dynamic-dynamodb - WARNING - sdtest - LimitExceededException: Subscriber limit exceeded: Provisioned throughput can be decreased only twice within the same day. Number of Decreases Today: 4. Last decrease time: Wednesday, January 22, 2014 3:54:00 PM UTC. Request time: Wednesday, January 22, 2014 6:49:34 PM UTC
2014-01-22 19:49:34,253 - dynamic-dynamodb - INFO - sdtest - Will retry to update provisioning with only increases
2014-01-22 19:49:36,757 - dynamic-dynamodb - DEBUG - sdtest - Currently provisioned read units: 7
2014-01-22 19:49:37,108 - dynamic-dynamodb - DEBUG - sdtest - Currently provisioned write units: 7
2014-01-22 19:49:37,109 - dynamic-dynamodb - INFO - sdtest - Retrying to update provisioning, excluding any decreases. Setting new reads to 7 and new writes to 7
2014-01-22 19:49:37,465 - dynamic-dynamodb - WARNING - sdtest - ValidationException: The provisioned throughput for the table will not change. The requested value equals the current value. Current ReadCapacityUnits provisioned for the table: 7. Requested ReadCapacityUnits: 7. Current WriteCapacityUnits provisioned for the table: 7. Requested WriteCapacityUnits: 7. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits.
@fwachsmuth

We deployed it earlier -- looking good! Ready to merge I'd say. Awesome and thanks for the speedy fix!

@sebdah
Owner
sebdah commented Jan 23, 2014

Perfect, thanks! Will merge and release later.

@sebdah
Owner
sebdah commented Jan 23, 2014

This has now been released to PyPI in version 1.7.2!

@sebdah sebdah closed this Jan 23, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment