New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CU increases fail if decreases fail due to exceeded limits #103

Closed
fwachsmuth opened this Issue Jan 22, 2014 · 4 comments

Comments

Projects
None yet
2 participants
@fwachsmuth

fwachsmuth commented Jan 22, 2014

When dynamic-dynamo intends to change read- and write-provisioning simultaneously where one increases and the other one decreases, the increase never happens due to exceeded decrease-limits.

In this case, the increase should still be executed as a "standalone" update call.

We just ran into massive throttlings in production due to this dead-lock situation.

@ghost ghost assigned sebdah Jan 22, 2014

sebdah added a commit that referenced this issue Jan 22, 2014

sebdah added a commit that referenced this issue Jan 22, 2014

sebdah added a commit that referenced this issue Jan 22, 2014

@sebdah

This comment has been minimized.

Owner

sebdah commented Jan 22, 2014

The code has been cleaned and restructured. This functionality is now ported to support Global Secondary Indexes as well.

@fwachsmuth, can you please test with the branch called release/1.7.2 and report back when done? I'll make a proper release of this fix after that.

I've tested this (but with out any load, so no actual increase took place) and it looks good. Here's my log output.

2014-01-22 19:49:34,253 - dynamic-dynamodb - WARNING - sdtest - LimitExceededException: Subscriber limit exceeded: Provisioned throughput can be decreased only twice within the same day. Number of Decreases Today: 4. Last decrease time: Wednesday, January 22, 2014 3:54:00 PM UTC. Request time: Wednesday, January 22, 2014 6:49:34 PM UTC
2014-01-22 19:49:34,253 - dynamic-dynamodb - INFO - sdtest - Will retry to update provisioning with only increases
2014-01-22 19:49:36,757 - dynamic-dynamodb - DEBUG - sdtest - Currently provisioned read units: 7
2014-01-22 19:49:37,108 - dynamic-dynamodb - DEBUG - sdtest - Currently provisioned write units: 7
2014-01-22 19:49:37,109 - dynamic-dynamodb - INFO - sdtest - Retrying to update provisioning, excluding any decreases. Setting new reads to 7 and new writes to 7
2014-01-22 19:49:37,465 - dynamic-dynamodb - WARNING - sdtest - ValidationException: The provisioned throughput for the table will not change. The requested value equals the current value. Current ReadCapacityUnits provisioned for the table: 7. Requested ReadCapacityUnits: 7. Current WriteCapacityUnits provisioned for the table: 7. Requested WriteCapacityUnits: 7. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits.
@fwachsmuth

This comment has been minimized.

fwachsmuth commented Jan 23, 2014

We deployed it earlier -- looking good! Ready to merge I'd say. Awesome and thanks for the speedy fix!

@sebdah

This comment has been minimized.

Owner

sebdah commented Jan 23, 2014

Perfect, thanks! Will merge and release later.

@sebdah

This comment has been minimized.

Owner

sebdah commented Jan 23, 2014

This has now been released to PyPI in version 1.7.2!

@sebdah sebdah closed this Jan 23, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment