ValueError: Unknown format code 'd' for object of type 'str' #10

Closed
fwachsmuth opened this Issue Apr 4, 2013 · 6 comments

Projects

None yet

2 participants

@fwachsmuth

It seems that dynamic-dynamo stumbles over certain usage ratios.
This only happens with certain tables, not sure what makes the difference.

Example output:


ubuntu@blah:~$ dynamic-dynamodb --dry-run -c PROD_HashKnownByAccount.conf 
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Consumed reads: 811
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Provisioned reads: 1000
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Read usage: 82%
2013-04-04 14:38:01,385 - dynamic-dynamodb - dryrun - INFO - Consumed writes: 15
2013-04-04 14:38:01,386 - dynamic-dynamodb - dryrun - INFO - Provisioned writes: 200
2013-04-04 14:38:01,386 - dynamic-dynamodb - dryrun - INFO - Write usage: 8%
Traceback (most recent call last):
  File "/usr/local/bin/dynamic-dynamodb", line 7, in <module>
    main.main()
  File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/main.py", line 174, in main
    dynamic_ddb.run()
  File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 148, in run
    self.ensure_proper_provisioning()
  File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 207, in ensure_proper_provisioning
    throughput['write_units']))
ValueError: Unknown format code 'd' for object of type 'str'

with this config:


# AWS region to use
region: us-east-1

# How often should Dynamic DynamoDB monitor changes (in seconds)
check-interval: 300

# -------------------------------------------------------------------------------
[table: PROD_HashKnownByAccount]
# -------------------------------------------------------------------------------
#
# Read provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
reads-upper-threshold: 100
reads-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 20
decrease-reads-with: 80

# Maximum and minimum read provisioning
# Dynamic DynamoDB will not provision any more or less reads than this
min-provisioned-reads: 10
max-provisioned-reads: 1500

#
# Write provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
writes-upper-threshold: 100
writes-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-writes-with: 30
decrease-writes-with: 60

# Maximum and minimum write provisioning
# Dynamic DynamoDB will not provision any more or less writes than this
min-provisioned-writes: 10
max-provisioned-writes: 500
@sebdah sebdah was assigned Apr 4, 2013
@sebdah
Owner
sebdah commented Apr 4, 2013

Thanks for the report. I have only looked briefly, but it should be a quick fix. Will look into it later today.

@fwachsmuth

Most awesome. I love dynamic-dynamo -- can't wait to be able to actually use it in production! :)

I observed that even with the same table it sometimes happens and sometimes not. Probably depending on the AWS responses somehow...?

@sebdah sebdah added a commit that referenced this issue Apr 4, 2013
@sebdah Converting str to int #10 2112a83
@sebdah sebdah closed this Apr 4, 2013
@sebdah
Owner
sebdah commented Apr 4, 2013

Thanks @fwachsmuth :). I believe that the bug has been properly addressed in version 0.3.1. You can upgrade your Dynamic DynamoDB with

pip install -U dynamic-dynamodb

Cheers again for the bug report!

@fwachsmuth

Thanks for the fix. It seems you broke something else though...

ubuntu@blah:~$ dynamic-dynamodb --dry-run -c PROD_HashKnownByAccount.conf
2013-04-05 07:17:21,876 - dynamic-dynamodb - dryrun - INFO - Consumed reads: 50
2013-04-05 07:17:21,876 - dynamic-dynamodb - dryrun - INFO - Provisioned reads: 1000
2013-04-05 07:17:21,876 - dynamic-dynamodb - dryrun - INFO - Read usage: 5%
2013-04-05 07:17:22,019 - dynamic-dynamodb - dryrun - INFO - Consumed writes: 12
2013-04-05 07:17:22,019 - dynamic-dynamodb - dryrun - INFO - Provisioned writes: 100
2013-04-05 07:17:22,019 - dynamic-dynamodb - dryrun - INFO - Write usage: 12%
2013-04-05 07:17:22,052 - dynamic-dynamodb - dryrun - INFO - Changing provisioning to 10 reads and 10 writes
2013-04-05 07:17:22,071 - dynamic-dynamodb - dryrun - INFO - Waiting 300s until checking again

The new CU-Provisiong of 10 reads and 10 writes looks way off to me.

The config file:


# AWS region to use
region: us-east-1

# How often should Dynamic DynamoDB monitor changes (in seconds)
check-interval: 300

# -------------------------------------------------------------------------------
[table: PROD_HashKnownByAccount]
# -------------------------------------------------------------------------------
#
# Read provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
reads-upper-threshold: 100
reads-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 20
decrease-reads-with: 80

# Maximum and minimum read provisioning
# Dynamic DynamoDB will not provision any more or less reads than this
min-provisioned-reads: 10
max-provisioned-reads: 1500

#
# Write provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
writes-upper-threshold: 100
writes-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-writes-with: 30
decrease-writes-with: 60

# Maximum and minimum write provisioning
# Dynamic DynamoDB will not provision any more or less writes than this
min-provisioned-writes: 10
max-provisioned-writes: 500
@fwachsmuth

Doesn't work in the other direction either:


2013-04-05 07:31:50,908 - dynamic-dynamodb - INFO - Consumed reads: 59
2013-04-05 07:31:50,908 - dynamic-dynamodb - INFO - Provisioned reads: 10
2013-04-05 07:31:50,908 - dynamic-dynamodb - INFO - Read usage: 590%
2013-04-05 07:31:50,996 - dynamic-dynamodb - INFO - Consumed writes: 8
2013-04-05 07:31:50,996 - dynamic-dynamodb - INFO - Provisioned writes: 10
2013-04-05 07:31:50,997 - dynamic-dynamodb - INFO - Write usage: 80%
2013-04-05 07:31:51,027 - dynamic-dynamodb - INFO - Changing provisioning to 12 reads and 10 writes
2013-04-05 07:31:51,231 - dynamic-dynamodb - INFO - Provisioning updated
2013-04-05 07:31:51,231 - dynamic-dynamodb - INFO - Waiting 300s until checking again

@sebdah
Owner
sebdah commented Apr 5, 2013

The last comments from @fwachsmuth is unrelated to #10 and has now been addressed in #13.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment