Use tornado AsyncHTTPClient in botocore
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
tornado_botocore Make curl dependency optional Oct 16, 2018
.flake8 v1.1.1, urparse.parse fro python3 May 17, 2017
.gitignore v1.1.1, urparse.parse fro python3 May 17, 2017
AUTHORS connect and request timeouts, None by default, version up to 1.4.0 Jun 8, 2018
CHANGELOG.txt
DESCRIPTION Fix up English a little bit. :) Oct 14, 2014
LICENSE added requests and pycurl to requirements, proxy support refactoring Feb 26, 2017
MANIFEST.in
README.rst Make curl dependency optional Oct 16, 2018
example.py updated botocore version May 21, 2017
requirements.txt Make curl dependency optional Oct 16, 2018
setup.py Make curl dependency optional Oct 16, 2018

README.rst

Tornado botocore

This module lets you use botocore with tornado's AsyncHTTPClient, so you can write asynchronous code in tornado for interacting with Amazon Web Services.

For async file upload to S3 see: https://gist.github.com/nanvel/c489761a11ec2db184c5

See also: https://github.com/qudos-com/botocore-tornado

Another option is aiohttp and aiobotocore: https://github.com/aio-libs/aiobotocore

And one another option, http client agnostic:

from types import MethodType

from botocore.endpoint import Endpoint
import botocore.session


class BotocoreRequest(Exception):

    def __init__(self, request, *args, **kwargs):
        super(BotocoreRequest, self).__init__(*args, **kwargs)
        self.method = request.method
        # https://github.com/twisted/treq/issues/185
        self.url = request.url.replace('https://', 'http://')
        self.headers = dict(request.headers)
        self.body = request.body and request.body.read()


def _send_request(self, request_dict, operation_model):
    request = self.create_request(request_dict, operation_model)
    raise BotocoreRequest(request=request)


class MyAWSClient:
    def __init__(self, service, access_key, secret_key, region, timeout=30):
        session = botocore.session.get_session()
        session.set_credentials(
            access_key=access_key,
            secret_key=secret_key
        )
        self.client = session.create_client(service, region_name=region)
        endpoint = self.client._endpoint
        endpoint._send_request = MethodType(_send_request, endpoint)
        self.timeout = timeout

    def request(self, method, **kwargs):
        try:
            getattr(self.client, method)(**kwargs)
        except BotocoreRequest as e:
            return MyFavouriteHTTPClient(
                method=e.method,
                url=e.url,
                body=e.body,
                headers=e.headers
            )

Installation

Requirements:
Versions:
  • tornado-botocore==0.0.3 (botocore==0.60.0)
  • tornado-botocore==0.1.0 (botocore==0.65.0)
  • tornado-botocore==1.0.0 (botocore==1.2)
  • tornado-botocore==1.2 (botocore>=1.2,<1.6)
  • tornado-botocore==1.4.0 (botocore 1.8+)
  • tornado-botocore==1.5.0 (botocore 1.12+)
pip install tornado-botocore

Example

A Simple EC2 Example from botocore docs:

import botocore.session


if __name__ == '__main__':
    session = botocore.session.get_session()
    client = session.create_client('ec2', region_name='us-west-2')

    for reservation in client.describe_instances()['Reservations']:
        for instance in reservation['Instances']:
            print(instance['InstanceId'])

Using tornado-botocore:

from tornado.ioloop import IOLoop
from tornado_botocore import Botocore


def on_response(response):
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            print(instance['InstanceId'])


if __name__ == '__main__':
    ec2 = Botocore(
        service='ec2',
        operation='DescribeInstances',
        region_name='us-east-1'
    )
    ec2.call(callback=on_response)
    IOLoop.instance().start()

If a callback is not specified, it works synchronously:

from tornado_botocore import Botocore


if __name__ == '__main__':
    ec2 = Botocore(
        service='ec2',
        operation='DescribeInstances',
        region_name='us-east-1'
    )
    print ec2.call()

Another example - deactivate SNS endpoint:

from tornado import gen
from tornado.ioloop import IOLoop
from tornado_botocore import Botocore


def on_response(response):
    print(response)
    # {'ResponseMetadata': {'RequestId': '056eb19e-3d2e-53e7-b897-fd176c3bb7f2'}}


if __name__ == '__main__':
    sns_operation = Botocore(
        service='sns',
        operation='SetEndpointAttributes',
        region_name='us-west-2'
    )
    sns_operation.call(
        callback=on_response,
        Endpoint='arn:aws:sns:us-west-2:...',
        Attributes={'Enabled': 'false'}
    )
    IOLoop.instance().start()

Send email using SES service and tonado.gen:

@gen.coroutine
def send(self, ...):
    ses_send_email = Botocore(
        service='ses',
        operation='SendEmail',
        region_name='us-east-1'
    )
    source = 'example@mail.com'
    message = {
        'Subject': {
            'Data': 'Example subject'.decode('utf-8'),
        },
        'Body': {
            'Html': {
                'Data': '<html>Example content</html>'.decode('utf-8'),
            },
            'Text': {
                'Data': 'Example content'.decode('utf-8'),
            }
        }
    }
    destination = {'ToAddresses': ['target@mail.com']}
    res = yield gen.Task(
        ses_send_email.call,
        Source=source,
        Message=message,
        Destination=destination
    )
    raise gen.Return(res)

Usage

Session: I think it makes sense to keep one global session object instead of create one for every request.

Credentials: You can specify credentials once on session object creation (pass to get_session method).

Testing: endpoint_url argument is useful for testing (use DynamoDBLocal).

Contribute

If you want to contribute to this project, please perform the following steps:

# Fork this repository
# Clone your fork
$ virtualenv .env --no-site-packages
$ source .env/bin/activate
$ pip install -r requirements.txt

$ git co -b feature_branch master
# Implement your feature
$ git add . && git commit
$ git push -u origin feature_branch
# Send us a pull request for your feature branch