Add support for temp. IAM credentials for S3 access #69

ckaenzig · 2014-02-21T14:29:44Z

These credentials are generated and rotated by Amazon for EC2
instances that have an instance role attributed. They are fetched
through the EC2 metadata API.

Note: this code is not fully tested yet and feedback about coding style is also welcome!

sbrunner · 2014-02-24T15:37:15Z

tilecloud/lib/s3.py

+        if self.iam_token is None \
+          or self.iam_token_expiration > datetime.utcnow() + timedelta(0, 300):
+            # Step 1: fetch the instance role name
+            api_connection = httplib.HTTPConnection('169.254.169.254')


From where does this IP com from ?

ckaenzig · 2014-02-24T16:47:30Z

Damn you tests! I will push a single commit once all the tests pass (and I manage to stop adding errors)

asaunier · 2014-03-05T13:28:35Z

Hi! I have tried to run this file in a project that uses S3 as tiles storage.
When running tilecloud-chain to generate a WMTS capabilities file

sudo -u www-data ./buildout/bin/generate_controller --capabilities

I get the following error stack:

Traceback (most recent call last):
  File "./buildout/bin/generate_controller", line 20, in <module>
    tilecloud_chain.controller.main()
  File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 85, in main
    _generate_wmts_capabilities(gene)
  File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 209, in _generate_wmts_capabilities
    _send(capabilities, cache['wmtscapabilities_file'], 'application/xml', cache)
  File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 104, in _send
    s3key.put()
  File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 128, in put
    return self.bucket.put(self.name, self.headers, self.body)
  File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 188, in put
    return self.connection.put(self.name, '/' + key_name, headers, body)
  File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 280, in put
    return self.request('PUT', bucket_name, url, body=body, headers=headers)
  File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 292, in request
    sub_resources)
  File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 347, in sign
    signature = hmac.new(self.secret_access_key, ''.join(string_to_sign), hashlib.sha1)
  File "/usr/lib/python2.7/hmac.py", line 133, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib/python2.7/hmac.py", line 72, in __init__
    self.outer.update(key.translate(trans_5C))
TypeError: character mapping must return integer, None or unicode

sbrunner · 2014-03-05T13:43:05Z

What's the relation with this pull request ?
you seem using an older egg 0.2dev_20131025 !

sbrunner · 2014-03-05T13:48:55Z

Or you copy the code in the existing egg ?

asaunier · 2014-03-05T13:53:42Z

Or you copy the code in the existing egg ?

Right!

sbrunner · 2014-03-05T13:59:29Z

tilecloud/lib/s3.py

+                                   % (response.status, response.reason))
+            credentials = json.loads(response.read())
+            self.access_key = credentials['AccessKeyId']
+            self.secret_access_key = credentials['SecretAccessKey']


@asaunier can you try to replace the upper line with:

print credentials['AccessKeyId'] print credentials['SecretAccessKey'] print type(credentials['AccessKeyId']) print type(credentials['SecretAccessKey']) self.secret_access_key = str(credentials['SecretAccessKey'])

I don't know if it is safe to paste here the access keys :P so I obfuscate them a bit:

A******************Q G********************************2 <type 'unicode'> <type 'unicode'>

and then same error ?

We should have a look together tomorrow it will be easier !

Oops, I forgot to add the str().
The error mentioned above no longer occurs but I get the original problem back:

Traceback (most recent call last): File "./buildout/bin/generate_controller", line 20, in <module> tilecloud_chain.controller.main() File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 85, in main _generate_wmts_capabilities(gene) File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 209, in _generate_wmts_capabilities _send(capabilities, cache['wmtscapabilities_file'], 'application/xml', cache) File "/home/alex/geoportal/buildout/eggs/tilecloud_chain-0.7.1-py2.7.egg/tilecloud_chain/controller.py", line 104, in _send s3key.put() File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 128, in put return self.bucket.put(self.name, self.headers, self.body) File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 188, in put return self.connection.put(self.name, '/' + key_name, headers, body) File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 285, in put return self.request('PUT', bucket_name, url, body=body, headers=headers) File "/home/alex/geoportal/buildout/eggs/tilecloud-0.2dev_20131025-py2.7.egg/tilecloud/lib/s3.py", line 305, in request raise S3Error(method, url, body, headers, response) tilecloud.lib.s3.S3Error: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

ckaenzig · 2014-03-05T14:44:36Z

According to this http://bugs.python.org/issue5285 (assuming it also applies to python 2.7), we should probably convert to the strings passed to hmac.new to String objects, one or both of them are probably Unicode now.

I'd happily take some time to check that out with you guys.

twpayne · 2014-03-24T18:56:53Z

@ckaenzig thanks for updating the pull request!

For info, you can run the tests locally with the command:

make pep8 pyflakes test

This is much quicker than waiting for Travis to run...

These credentials are generated and rotated by Amazon for EC2 instances that have an instance role attributed. They are fetched through the EC2 metadata API.

twpayne · 2014-03-25T13:44:00Z

Many thanks for this nice functionality!

Add support for temp. IAM credentials for S3 access

sbrunner reviewed Feb 24, 2014
View reviewed changes

sbrunner reviewed Mar 5, 2014
View reviewed changes

Add support for temp. IAM credentials for S3 access

9e077cc

These credentials are generated and rotated by Amazon for EC2 instances that have an instance role attributed. They are fetched through the EC2 metadata API.

twpayne added a commit that referenced this pull request Mar 25, 2014

Merge pull request #69 from ckaenzig/s3-temp-iam-credentials

7d69504

Add support for temp. IAM credentials for S3 access

twpayne merged commit 7d69504 into camptocamp:master Mar 25, 2014

ckaenzig deleted the s3-temp-iam-credentials branch March 25, 2014 14:06

sbrunner added this to the 0.3 milestone Apr 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for temp. IAM credentials for S3 access #69

Add support for temp. IAM credentials for S3 access #69

ckaenzig commented Feb 21, 2014

sbrunner Feb 24, 2014

ckaenzig commented Feb 24, 2014

asaunier commented Mar 5, 2014

sbrunner commented Mar 5, 2014

sbrunner commented Mar 5, 2014

asaunier commented Mar 5, 2014

sbrunner Mar 5, 2014

asaunier Mar 5, 2014

sbrunner Mar 5, 2014

sbrunner Mar 5, 2014

asaunier Mar 5, 2014

ckaenzig commented Mar 5, 2014

twpayne commented Mar 24, 2014

twpayne commented Mar 25, 2014

Add support for temp. IAM credentials for S3 access #69

Add support for temp. IAM credentials for S3 access #69

Conversation

ckaenzig commented Feb 21, 2014

sbrunner Feb 24, 2014

Choose a reason for hiding this comment

ckaenzig commented Feb 24, 2014

asaunier commented Mar 5, 2014

sbrunner commented Mar 5, 2014

sbrunner commented Mar 5, 2014

asaunier commented Mar 5, 2014

sbrunner Mar 5, 2014

Choose a reason for hiding this comment

asaunier Mar 5, 2014

Choose a reason for hiding this comment

sbrunner Mar 5, 2014

Choose a reason for hiding this comment

sbrunner Mar 5, 2014

Choose a reason for hiding this comment

asaunier Mar 5, 2014

Choose a reason for hiding this comment

ckaenzig commented Mar 5, 2014

twpayne commented Mar 24, 2014

twpayne commented Mar 25, 2014