Initial version of BigMessage.

[garnaat]

This is an initial attempt at a big message, basically an SQS message that stores it's payload in S3 and therefore allows for arbitrarily large payloads.

An example:

    import boto.sqs
    from boto.sqs.bigmessage import BigMessage

    sqs = boto.sqs.connect_to_region('us-west-2')
    queue = sqs.get_queue('myqueue')
    fp = open('/path/to/bigmessage/data')
    msg = BigMessage(queue, fp, 's3://mybucket')
    queue.write(msg)

You need to pass in a file-like object as the body of the message. You also have to pass in an s3_url to specify where the message should be stored. The s3_url should be of the form:

s3://bucket[/key]

If no key is specified, a key will be created using UUID's and the body of the message will be stored in the specified bucket using the UUID as a key. The contents of the file-like object are uploaded to that key in S3 and a reference to the new object is stored in the message body. If a bucket and key are specified, it is interpreted as specifying an existing object in S3 and no object is uploaded. The BigMessage will simply refer to the existing object in the body of the message.

When a BigMessage object is deleted, it will attempt to delete the object in S3.

Feedback on the overall operability of this implementation would be appreciated as well as feedback on the code itself.

[toastdriven]

Should we say "up to 5Gb" (the S3 limit) here? It's unlikely someone will do it, but they should be aware & it'll give people a sense of scale.

[garnaat]

Yep.

[toastdriven]

LGTM otherwise.

[garnaat]

Note that this would fix #1902.

cc @jamesls, @diranged

[diranged]

Excellent... I'm going to look into how to integrate this into Kombu/Celery ASAP. I'll do a more thorough review a little later today just so you have another set of eyes on this change.

[danielgtaylor]

Other than the comment above this looks good 👍

[diranged]

Shouldn't you try to delete the S3 file first? If the RawMessage.delete(self) raises any exception, then the file is left over dangling in S3 land forever.

[danielgtaylor]

Latest changes look good. 🚢-it