90%: Queue bulk composite generation #55
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -d --spec Only create for specified search document specs | ||
| -i --id Resource ID to regenerate search documents for | ||
| -a --async Generate table rows via queue | ||
| -q --queue Queue name to place jobs on (defaults to configured TRIPOD_APPLY_QUEUE value) |
There was a problem hiding this comment.
If you specify an alternative queue, how do you know anything is listening to it?
There was a problem hiding this comment.
You don't - the purpose is you'd fire up a worker to listen for that queue to keep it from polluting/clogging the main queue
|
👍 |
rsinger
added a commit
that referenced
this pull request
May 22, 2015
90%: Queue bulk composite generation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This refactors the createTables and createViews scripts to allow them to push the actual regeneration to the resque queue. You may also specify which queue you want these to go on, so as not to clog the regular process queue.
It also introduces a createSearchDocuments script which never existed in Tripod.
The major difference between these and how they work currently is that the current version loops over the CBDs and generates the composites serially. This version gives the option to loop over the CBDs and submits a job to generate the composite, so the actual generation can be scaled/stopped as needed without starting from the beginning to resume.
If you don't pass the -a/--async flag it works exactly as it does now.