This repository has been archived by the owner on Aug 26, 2022. It is now read-only.
Bug 1125850 - Optimize sitemap celery task #3290
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jezdez
force-pushed
the
bug1125850-6
branch
2 times, most recently
from
23f42c0
to
cefa731
Compare
|
|
||
| del queryset # Force the gc to cleanup | ||
| for result in results: | ||
| # result can be empty of no documents were found |
|
Looks great. r+wc |
This does a bunch of things: - limit the documents we want to write sitemap files for (by using an already existing model manager) - stop fetching the html blob from the database - split the sitemap building into tasks per locale to distribute load to all celery nodes - use a chord to write the sitemap index after building all individual sitemap files - write sitemap files that are above the 50k items per sitemap limit instead of just the first one
jezdez
added a commit
that referenced
this pull request
Jul 1, 2015
Bug 1125850 - Optimize sitemap celery task
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This does a bunch of things: