Using DynamoDB with IAM Roles gives "403 Forbidden" when trying to connect #830

Closed
jonathanq opened this Issue Jun 20, 2012 · 2 comments

Projects

None yet

2 participants

When creating a DynamoDB connection using EC2 IAM Roles we get a 403 Forbidden error when calling connect_dynamodb()

import boto
conn = boto.connect_dynamodb()
conn.list_tables

And this is the error we get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/__init__.py", line 563, in connect_dynamodb
    return Layer2(aws_access_key_id, aws_secret_access_key, **kwargs)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer2.py", line 111, in __init__
    debug, session_token, region)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 96, in __init__
    session_token = self._get_session_token()
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 120, in _get_session_token
    return sts.get_session_token()
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/sts/connection.py", line 107, in get_session_token
    token = self._get_session_token(duration)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/sts/connection.py", line 77, in _get_session_token
    Credentials, verb='POST')
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/connection.py", line 967, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 403 Forbidden

It appears that boto is trying to use the IAM temp credentials to create a session token - and it is failing.

A workaround we came up with is to use the session token that already exists in the IAM role instance metadata:

# Get IAM role data and force it through when boto attempts to make the Dynamo connection,
# else it will try to create a new STS token and fail
from boto.sts.credentials import Credentials
metadata = boto.utils.get_instance_metadata()
if metadata and 'iam' in metadata:
    iam_role_data = metadata['iam']['security-credentials'].values()[0] # Currently one role per instance
    credentials = Credentials()
    credentials.access_key = iam_role_data['AccessKeyId']
    credentials.secret_key = iam_role_data['SecretAccessKey']
    credentials.session_token = iam_role_data['Token']
    credentials.expiration = iam_role_data['Expiration']
    return boto.connect_dynamodb(session_token=credentials)
else:
    return boto.connect_dynamodb()
Owner
garnaat commented Jun 21, 2012

Thanks for reporting. This is definitely a boto bug and I'm in the process of fixing it.

I'm curious what happens in your workaround when the credentials for the IAM Role that you are pulling out of the metadata expire. By default, that happens in an hour.

I can tell you exactly what happens :) This workaround fails. The _retry_handler will go back and try to create a new session token using the temp credentials and it fails.

So I guess I spoke too soon with the workaround.

  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/table.py", line 184, in get_item
    item_class)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer2.py", line 412, in get_item
    object_hook=item_object_hook)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 310, in get_item
    object_hook=object_hook)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 140, in make_request
    retry_handler=self._retry_handler)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/connection.py", line 771, in _mexe
    status = retry_handler(response, i, next_sleep)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 167, in _retry_handler
    self.creds = self._get_session_token()
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 120, in _get_session_token
    return sts.get_session_token()
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/sts/connection.py", line 107, in get_session_token
    token = self._get_session_token(duration)
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/sts/connection.py", line 77, in _get_session_token
    Credentials, verb='POST')
  File "/home/ec2-user/myapp/pyenv/lib/python2.7/site-packages/boto/connection.py", line 966, in get_object
    raise self.ResponseError(response.status, response.reason, body)
  BotoServerError: BotoServerError: 403 Forbidden
@garnaat garnaat closed this in 842a6a1 Jun 21, 2012
@msabramo msabramo pushed a commit to msabramo/boto that referenced this issue Nov 28, 2012
@garnaat garnaat Fix to allow DynamoDB to operate properly with IAM Roles on EC2 insta…
…nces. Fixes #830.
a03c3c5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment