AWS-ome backend #68
Merged
AWS-ome backend #68
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
autoload has/had thread issues. Requiring worker is fine.
The payload that is passed to release is always the payload that is returned by reserve. This allows backends to tack on a bit more info for use in release and completed.
Uses SNS or SQS to enqueue items, SNS to dequeue, and Dynamo to store state.
mock is deprecated.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See what I did there. Yeaaaaaahhhhh.
This adds an AWS backend and is the first step towards a better abstraction that I am hoping to work on. The AWS Connection makes it possible to use SNS or SQS to enqueue jobs and SQS to dequeue. All state of known queues and workers is stored in dynamodb.
The abstraction I want to change is from "backend" to "state" and "queue". Right now, the requirements of a "backend" require storing arbitrary things (workers, queues) that all of the real and great queues don't support. Also, most of this arbitrary information is used only for UI purposes (of which there is not one currently).
Separating backend into state and queue means that all queues can work with qu and one can store the UI state where it makes more sense (mysql, redis, mongo, dynamo, simpledb, whatever) or not at all. Hoping to work on that next, but need to power through the AWS stuff in order to see what was needed.
Also, this adds the idea of required services for a backend spec. It means if you provide services, we'll make sure they are up when the specs run and if they aren't we'll skip those specs so you dont have to have every queue and database in the world running with exactly the right options all the time to work on qu.