No error message when having insufficient IAM permissions #166

Closed
ulsa opened this Issue May 16, 2014 · 16 comments

Projects

None yet

2 participants

@ulsa
Contributor
ulsa commented May 16, 2014

I start it as a daemon like this:

sudo dynamic-dynamodb -c /etc/example/dynamic-dynamodb.conf --daemon start --pid-file-dir /var/run

without any errors, but no scaling is performed and the log just says this:

2014-05-16 15:05:20,934 - dynamic-dynamodb - DEBUG - Authenticating to SNS using EC2 instance profile
2014-05-16 15:05:20,976 - dynamic-dynamodb - DEBUG - Connected to SNS in eu-west-1
2014-05-16 15:05:20,977 - dynamic-dynamodb - DEBUG - Connecting to DynamoDB in eu-west-1
2014-05-16 15:05:20,978 - dynamic-dynamodb - DEBUG - Authenticating to DynamoDB using EC2 instance profile
2014-05-16 15:05:20,988 - dynamic-dynamodb - DEBUG - Connected to DynamoDB in eu-west-1
2014-05-16 15:05:21,095 - dynamic-dynamodb - DEBUG - Authenticating to CloudWatch using EC2 instance profile
2014-05-16 15:05:21,106 - dynamic-dynamodb - DEBUG - Connected to CloudWatch in eu-west-1
2014-05-16 15:05:21,452 - dynamic-dynamodb - DEBUG - Table dev-OneDynamoDBTable-1NZQE9XTDXD81 match with config key ^dev-.+DynamoDBTable-.+$
2014-05-16 15:05:21,452 - dynamic-dynamodb - DEBUG - Table dev-TwoDynamoDBTable-TIC25BS7EPSE match with config key ^dev-.+DynamoDBTable-.+$
2014-05-16 15:05:21,452 - dynamic-dynamodb - DEBUG - Table dev-ThreeDynamoDBTable-1TL5MPJ9HNBSN match with config key ^dev-.+DynamoDBTable-.+$
2014-05-16 15:05:21,452 - dynamic-dynamodb - DEBUG - Table dev-FourDynamoDBTable-1LHSXXAPI9TRS match with config key ^dev-.+DynamoDBTable-.+$
2014-05-16 15:05:21,483 - dynamic-dynamodb - DEBUG - dev-OneDynamoDBTable-1NZQE9XTDXD81 - Currently provisioned read units: 10

Process is running:

$ ps -ef|grep dynamic
root     25731     1  0 17:51 ?        00:00:00 /usr/bin/python /usr/local/bin/dynamic-dynamodb -c /etc/example/dynamic-dynamodb.conf --daemon start --pid-file-dir /var/run
@sebdah sebdah added this to the 1.14.x milestone May 19, 2014
@sebdah sebdah self-assigned this May 19, 2014
@sebdah sebdah removed this from the 1.14.x milestone May 19, 2014
@sebdah
Owner
sebdah commented May 19, 2014

Thanks for the report, will try to reproduce this and investigate.

@sebdah sebdah added this to the 1.14.x milestone May 19, 2014
@sebdah
Owner
sebdah commented May 19, 2014

I cannot reproduce this, can you please submit your configuration file and dynamic-dynamodb version number?

@ulsa
Contributor
ulsa commented May 19, 2014
Dynamic DynamoDB version: 1.13.1
[global]

# AWS region to use
region: eu-west-1

# How often should Dynamic DynamoDB monitor changes (in seconds)
check-interval: 300

[logging]
# Log level [debug|info|warning|error]
log-level: debug

# Log file (comment out to get only console output)
log-file: /var/log/dynamic-dynamodb.log

# External Python logging configuration file
# Overrides both log-level and log-file
# log-config-file: /path/to/logging.conf

[table: ^dev-.+DynamoDBTable-.+$]
#
# Read provisioning configuration
#

# Enable or disable reads autoscaling
enable-reads-autoscaling = true

# Thresholds for scaling up or down the provisioning (%)
reads-upper-threshold: 90
reads-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 50
decrease-reads-with: 50

# Units to increase or decrease reads with, must be either percent or units
increase-reads-unit: percent
decrease-reads-unit: percent

# Maximum and minimum read provisioning
# Dynamic DynamoDB will not provision any more or less reads than this
min-provisioned-reads: 1
max-provisioned-reads: 500

#
# Write provisioning configuration
#

# Enable or disable writes autoscaling
enable-writes-autoscaling = true

# Thresholds for scaling up or down the provisioning (%)
writes-upper-threshold: 90
writes-lower-threshold: 30

# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-writes-with: 50
decrease-writes-with: 50

# Units to increase or decrease writes with, must be either percent or units
increase-writes-unit: percent
decrease-writes-unit: percent

# Maximum and minimum write provisioning
# Dynamic DynamoDB will not provision any more or less writes than this
min-provisioned-writes: 1
max-provisioned-writes: 1000

# Other settings
#

# Allow down scaling when at 0% consumed reads
#allow-scaling-down-reads-on-0-percent: true
#allow-scaling-down-writes-on-0-percent: true

# Restrict scale down to only happen when BOTH reads AND writes are in need
# of scaling down. Set this to "true" to minimize down scaling.
#always-decrease-rw-together: true
@sebdah
Owner
sebdah commented May 20, 2014

I've been trying this with your config file (but with credentials from my env vars) and the same dynamic-dynamodb version, but I can't reproduce the error. I have a hard time seeing what the reason might be. Have you tried running it without daemon mode?

@ulsa
Contributor
ulsa commented May 20, 2014

Yes, tried with and without daemon, and also with default /tmp for pids, in case there were some permission problems. No luck. It says:

dev-OneDynamoDBTable-1NZQE9XTDXD81 - Currently provisioned read units: 10

and then nothing.

Have you tried with four tables using the same regexp?

@ulsa
Contributor
ulsa commented May 20, 2014

Tried command-line too, with only one table. Same problem, nothing happens after first table's provisioning is read:

$ sudo dynamic-dynamodb --table dev-OneDynamoDBTable-1NZQE9XTDXD81 --reads-upper-threshold 90 --reads-lower-threshold 30 --increase-reads-with 50 --decrease-reads-with 50 --min-provisioned-reads 10 --max-provisioned-reads 500 --writes-upper-threshold 90 --writes-lower-threshold 30 --increase-writes-with 50 --decrease-writes-with 50 --min-provisioned-writes 10 --max-provisioned-writes 1000 --check-interval 300 --log-file /var/log/dynamic-dynamodb.log --log-level debug
...
2014-05-20 07:24:20,567 - dynamic-dynamodb - DEBUG - dev-OneDynamoDBTable-1NZQE9XTDXD81 - Currently provisioned read units: 10
@ulsa
Contributor
ulsa commented May 20, 2014

Noticed something weird that might be unrelated to this problem. Forgot to set region on command-line, so all connections were to us-east-1. But it still said my table name matched, it could read all four table names from AWS, and it managed to get the read provisioning for the matching table. That shouldn't have worked, right?

$ sudo dynamic-dynamodb --table dev-OneDynamoDBTable-1NZQE9XTDXD81 ...
2014-05-20 07:24:20,309 - dynamic-dynamodb - DEBUG - Connected to SNS in us-east-1
2014-05-20 07:24:20,349 - dynamic-dynamodb - DEBUG - Connected to DynamoDB in us-east-1
2014-05-20 07:24:20,485 - dynamic-dynamodb - DEBUG - Connected to CloudWatch in us-east-1
2014-05-20 07:24:20,549 - dynamic-dynamodb - DEBUG - Table dev-OneDynamoDBTable-1NZQE9XTDXD81 match with config key dev-OneDynamoDBTable-1NZQE9XTDXD81
2014-05-20 07:24:20,550 - dynamic-dynamodb - DEBUG - Table dev-TwoDynamoDBTable-TIC25BS7EPSE did not match with config key dev-OneDynamoDBTable-1NZQE9XTDXD81
2014-05-20 07:24:20,550 - dynamic-dynamodb - DEBUG - Table dev-ThreeDynamoDBTable-1TL5MPJ9HNBSN did not match with config key dev-OneDynamoDBTable-1NZQE9XTDXD81
2014-05-20 07:24:20,551 - dynamic-dynamodb - DEBUG - Table dev-FourDynamoDBTable-1LHSXXAPI9TRS did not match with config key dev-OneDynamoDBTable-1NZQE9XTDXD81
2014-05-20 07:24:20,567 - dynamic-dynamodb - DEBUG - dev-OneDynamoDBTable-1NZQE9XTDXD81 - Currently provisioned read units: 10
@sebdah
Owner
sebdah commented May 20, 2014

I have tested a million combinations but can't reproduce the original error you had. I always get past the Currently provisioned read units point. I do not think it is related to you other issue.

The second issue you have is a bug in Dynamic DynamoDB, will open a new issue for that. I assume that you are using instance profiles?

@ulsa
Contributor
ulsa commented May 20, 2014

Yes, I use instance profiles. Perhaps there are some permissions lacking? Which ones are required?

@sebdah
Owner
sebdah commented May 20, 2014
@ulsa
Contributor
ulsa commented May 20, 2014

I thought I had cloudwatch:*, but it turned out I had just PutMetricData. Let me fix and verify.

It kind of makes sense, getting the current table provisioning, then hanging waiting for metrics and not getting any. Could you perhaps catch that situation and warn?

@sebdah
Owner
sebdah commented May 20, 2014

Yeah. I did expect at least an unhandled exception for that. Will have to test a bit before implementing it.

Let me know if that solved the issue.

@ulsa
Contributor
ulsa commented May 20, 2014

That solved it.

@sebdah sebdah changed the title from Nothing happens, and no error messages to No error message when having insufficient IAM permissions May 20, 2014
@sebdah
Owner
sebdah commented May 20, 2014

Ok, thanks. I updated the issue title and will look at a fix for that.

@ulsa
Contributor
ulsa commented May 20, 2014

I assume SNS permissions are only needed if I use the sns options, but could you perhaps list what they are in the docs/readme?

@sebdah
Owner
sebdah commented May 20, 2014

Added #171 to fix that.

@sebdah sebdah closed this May 21, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment