Question: Dynamic-DynamoDB does not seem to be scaling down. Improper setting? #142

Closed
JasonNethercott opened this Issue Mar 31, 2014 · 12 comments

Projects

None yet

2 participants

@JasonNethercott

Log:

    2014-03-31 12:58:51,617 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
    2014-03-31 12:58:51,683 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
    2014-03-31 12:58:51,683 - dynamic-dynamodb - INFO - myTestTable - Consumed read units: 0%
    2014-03-31 12:58:51,702 - dynamic-dynamodb - INFO - myTestTable - Read throttle count: 0
    2014-03-31 12:58:51,719 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
    2014-03-31 12:58:51,779 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
    2014-03-31 12:58:51,779 - dynamic-dynamodb - INFO - myTestTable - Consumed write units: 0%
    2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - Write throttle count: 0
    2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - No need to change provisioning

Settings:

    [table: myTestTable]
    reads-upper-threshold: 90
    reads-lower-threshold: 30

    increase-reads-with: 50
    decrease-reads-with: 50

    increase-reads-unit: percent
    decrease-reads-unit: percent

    min-provisioned-reads: 5
    max-provisioned-reads: 25

    writes-upper-threshold: 90
    writes-lower-threshold: 30

    increase-writes-with: 50
    decrease-writes-with: 50

    increase-writes-unit: percent
    decrease-writes-unit: percent

    min-provisioned-writes: 5
    max-provisioned-writes: 100

    #maintenance-windows: 22:00-23:59,00:00-06:00

    sns-message-types: scale-up, scale-down

    allow-scaling-down-reads-on-0-percent: true
    allow-scaling-down-writes-on-0-percent: true

    #always-decrease-rw-together: true

Scaling up seems to be working, just not down. I've let it run overnight but it just does not want to downscale.

Thanks

@sebdah
Owner
sebdah commented Mar 31, 2014

It looks like a bug in the throttling feature. I have corrected this and will soon release a patch.

@sebdah sebdah added this to the 1.10.x milestone Mar 31, 2014
@sebdah sebdah self-assigned this Mar 31, 2014
@sebdah
Owner
sebdah commented Mar 31, 2014

Thanks for the bug report. I was able to reproduce the issue and dynamic-dynamodb 1.10.4 is released which should fix this.

@sebdah
Owner
sebdah commented Mar 31, 2014

Please let me know when you have confirmed that this fixes the issue for you. I will close the issue for now, but please feel free to re-open it if the problem persists.

@sebdah sebdah closed this Mar 31, 2014
@JasonNethercott

I will test it today...

@JasonNethercott

I've run a "pip install --upgrade..." It tells me what version was found and upgraded, but not which it was upgraded to. Is there a command to tell me what version I have running?

@sebdah
Owner
sebdah commented Mar 31, 2014

cumulus --version

Sebastian Dahlgren

sebastian dahlgren | sebastian.dahlgren@gmail.com | +46 760 19 42 40 |
www.sebastiandahlgren.se | skype: sebastian.dahlgren | gtalk:
sebastian.dahlgren@gmail.com

On Mon, Mar 31, 2014 at 7:42 AM, Jason Nethercott
notifications@github.comwrote:

I've run a "pip install --upgrade..." It tells me what version was found
and upgraded, but not which it was upgraded to. Is there a command to tell
me what version I have running?

Reply to this email directly or view it on GitHubhttps://github.com/sebdah/dynamic-dynamodb/issues/142#issuecomment-39096160
.

@JasonNethercott

Initial testing shows that it's down scaling the reads... but not the writes. I do have activity in my environment, so I'm continuing to test

@sebdah sebdah reopened this Mar 31, 2014
@sebdah
Owner
sebdah commented Mar 31, 2014

Thanks. Yeah, you're right, writes had the same problem global secondary indexes. Both fixed in 1.10.5.

@JasonNethercott

Is it just the GSIs? Or the tables too?

@sebdah
Owner
sebdah commented Mar 31, 2014

It was both, but should be fixed for both now.

Sebastian Dahlgren
On Mar 31, 2014 5:23 PM, "Jason Nethercott" notifications@github.com
wrote:

Is it just the GSIs? Or the tables too?

Reply to this email directly or view it on GitHubhttps://github.com/sebdah/dynamic-dynamodb/issues/142#issuecomment-39101369
.

@JasonNethercott

upgraded to 1.10.5 and it looks much better on initial start up. The write provisioning was all cut in 1/2 because I paused all processing.

Question: How does dynamic-dynamodb handle the: "can only down scale a table provisioning 4x a day" restriction? I assume, if it exceeds the limit it will keep trying (every 300 seconds) until the provisioning change is accepted?

@sebdah
Owner
sebdah commented Mar 31, 2014

Perfect!

Yeah, you're right. We will keep trying until it is accepted. There are four options that can help you target the 4 times / UTC day when it is most valuable to scale down:

  • reads-lower-threshold
  • writes-lower-threshold
  • maintenance-windows
  • always-decrease-rw-together
@sebdah sebdah closed this Mar 31, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment