Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add support for an SQS Message class backed by S3 #1902

Open
jamesls opened this Issue Dec 11, 2013 · 3 comments

Comments

Projects
None yet
3 participants
Owner

jamesls commented Dec 11, 2013

There have been requests to transparently handle large message bodies
by saving the message body to S3. When a message is received from SQS,
the message body will be downloaded from S3.

We should be able to implement with this a custom Message class,
and possibly a custom Queue class. You could then change the
defaut message/queue classes over to this variant if you wanted
this behavior, but we wouldn't change any default values.

I'm filing this issue to help track this feature. If anyone has any
thoughts/feedback on this, please feel free to share.

cc @diranged

@jamesls thanks for opening this issue. Indeed, I think that you could have support for S3 as a temporary message store for large messages. Taking this further, you could also allow for using DynamoDB for messages that are > 256KB < 1MB. For some users, this may be faster and more desirable than using S3.

Overall this feature seems to be not terribly complicated... and as soon as its completed I'll be happy to add support to https://github.com/celery/kombu.

Owner

garnaat commented Dec 12, 2013

I think DynamoDB limits the size of any item to be 64K total.

Ah ... sorry, could have sworn that Dynamo had a larger message size limit.

@ghost ghost assigned garnaat Dec 18, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment