Your Friendly Asynchronous S3 Upload Protocol Droid
Pull request Compare This branch is 69 commits behind seomoz:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Your friendly neighborhood S3 helper. It knows just a few tricks, but hopefully they'll help you out.

  • Automatic retries -- with exponential backoff to make your life easier
  • Parallel batch operations -- using gevent to make it hella fast
  • Built-in S3 Mocking -- to make your testing eaiser
  • Automatic multipart uploading -- and it can retry each piece individually


This module requires that you have boto installed. After that:

sudo python install

Basic Use

s3po knows a few tricks, but at its core, you'll use two methods: upload and download:

import s3po
conn = s3po.Connection()

# Upload with a string
conn.upload('bucket', 'key', 'howdy')
# Upload a file
with open('foo.txt') as fin:
    conn.upload('bucket', 'key', fin)

# Download as a string
print'bucket', 'key')
# Or into a file
with open('foo.txt', 'w') as fout:'bucket', 'key', fout)

Batch Use

The batch object works like a context manager that provide the same interface as the connection object. The only difference is that all the requests run in a gevent pool. When the context is closed, it waits for all functions to finish.

# Upload these in a gevent pool
keys = ['key.%i' for key in range(1000)]
with conn.batch(50) as batch:
    for key in keys:
        batch.upload('bucket', key, 'hello')

# And now they're all uploaded


When you want to take some action with the result, all the functions have an additional callback optional parameter to you can grab the result. This is particularly useful when doing bulk downloads:

from functools import partial

def func(key, value):
    print 'Downloaded %s from %s' % (key, value)

with conn.batch(50) as batch:
    for key in keys:'bucket', key, callback=partial(func, key))


If the provided data is sufficiently large, it will automatically run the upload as a multipart upload rather than a single upload. The advantage here is that for any part upload that fails, it will retry just that part.


You can turn on mocking to get the same functionality of s3po that you'd expect but without ever having to touch S3. Use it as a context manager:

# This doesn't touch S3 at all
with conn.mock():
    conn.upload('foo', 'bar', 'hello')

If you're writing tests, this is a common pattern:

class MyTest(unittest.TestCase):
    def setUp(self):
        self.s3po = Connectino()
        self.mock = self.s3po.mock()

    def tearDown(self):