New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: Dynamic-DynamoDB does not seem to be scaling down. Improper setting? #142

Closed
JasonNethercott opened this Issue Mar 31, 2014 · 12 comments

Comments

Projects
None yet
2 participants
@JasonNethercott

JasonNethercott commented Mar 31, 2014

Log:

    2014-03-31 12:58:51,617 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
    2014-03-31 12:58:51,683 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
    2014-03-31 12:58:51,683 - dynamic-dynamodb - INFO - myTestTable - Consumed read units: 0%
    2014-03-31 12:58:51,702 - dynamic-dynamodb - INFO - myTestTable - Read throttle count: 0
    2014-03-31 12:58:51,719 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
    2014-03-31 12:58:51,779 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
    2014-03-31 12:58:51,779 - dynamic-dynamodb - INFO - myTestTable - Consumed write units: 0%
    2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - Write throttle count: 0
    2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - No need to change provisioning

Settings:

    [table: myTestTable]
    reads-upper-threshold: 90
    reads-lower-threshold: 30

    increase-reads-with: 50
    decrease-reads-with: 50

    increase-reads-unit: percent
    decrease-reads-unit: percent

    min-provisioned-reads: 5
    max-provisioned-reads: 25

    writes-upper-threshold: 90
    writes-lower-threshold: 30

    increase-writes-with: 50
    decrease-writes-with: 50

    increase-writes-unit: percent
    decrease-writes-unit: percent

    min-provisioned-writes: 5
    max-provisioned-writes: 100

    #maintenance-windows: 22:00-23:59,00:00-06:00

    sns-message-types: scale-up, scale-down

    allow-scaling-down-reads-on-0-percent: true
    allow-scaling-down-writes-on-0-percent: true

    #always-decrease-rw-together: true

Scaling up seems to be working, just not down. I've let it run overnight but it just does not want to downscale.

Thanks

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

It looks like a bug in the throttling feature. I have corrected this and will soon release a patch.

@sebdah sebdah added this to the 1.10.x milestone Mar 31, 2014

@sebdah sebdah self-assigned this Mar 31, 2014

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

Thanks for the bug report. I was able to reproduce the issue and dynamic-dynamodb 1.10.4 is released which should fix this.

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

Please let me know when you have confirmed that this fixes the issue for you. I will close the issue for now, but please feel free to re-open it if the problem persists.

@sebdah sebdah closed this Mar 31, 2014

@JasonNethercott

This comment has been minimized.

JasonNethercott commented Mar 31, 2014

I will test it today...

@JasonNethercott

This comment has been minimized.

JasonNethercott commented Mar 31, 2014

I've run a "pip install --upgrade..." It tells me what version was found and upgraded, but not which it was upgraded to. Is there a command to tell me what version I have running?

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

cumulus --version

Sebastian Dahlgren

sebastian dahlgren | sebastian.dahlgren@gmail.com | +46 760 19 42 40 |
www.sebastiandahlgren.se | skype: sebastian.dahlgren | gtalk:
sebastian.dahlgren@gmail.com

On Mon, Mar 31, 2014 at 7:42 AM, Jason Nethercott
notifications@github.comwrote:

I've run a "pip install --upgrade..." It tells me what version was found
and upgraded, but not which it was upgraded to. Is there a command to tell
me what version I have running?

Reply to this email directly or view it on GitHubhttps://github.com//issues/142#issuecomment-39096160
.

@JasonNethercott

This comment has been minimized.

JasonNethercott commented Mar 31, 2014

Initial testing shows that it's down scaling the reads... but not the writes. I do have activity in my environment, so I'm continuing to test

sebdah added a commit that referenced this issue Mar 31, 2014

@sebdah sebdah reopened this Mar 31, 2014

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

Thanks. Yeah, you're right, writes had the same problem global secondary indexes. Both fixed in 1.10.5.

@JasonNethercott

This comment has been minimized.

JasonNethercott commented Mar 31, 2014

Is it just the GSIs? Or the tables too?

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

It was both, but should be fixed for both now.

Sebastian Dahlgren
On Mar 31, 2014 5:23 PM, "Jason Nethercott" notifications@github.com
wrote:

Is it just the GSIs? Or the tables too?

Reply to this email directly or view it on GitHubhttps://github.com//issues/142#issuecomment-39101369
.

@JasonNethercott

This comment has been minimized.

JasonNethercott commented Mar 31, 2014

upgraded to 1.10.5 and it looks much better on initial start up. The write provisioning was all cut in 1/2 because I paused all processing.

Question: How does dynamic-dynamodb handle the: "can only down scale a table provisioning 4x a day" restriction? I assume, if it exceeds the limit it will keep trying (every 300 seconds) until the provisioning change is accepted?

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 31, 2014

Perfect!

Yeah, you're right. We will keep trying until it is accepted. There are four options that can help you target the 4 times / UTC day when it is most valuable to scale down:

  • reads-lower-threshold
  • writes-lower-threshold
  • maintenance-windows
  • always-decrease-rw-together
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment