Improve publish speed by between 2x and 20x #493
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dralley
force-pushed
the
publish-improvement
branch
from
28d24d8
to
960b818
Compare
|
Attached issue: https://pulp.plan.io/issues/8508 |
dralley
force-pushed
the
publish-improvement
branch
3 times, most recently
from
13c1040
to
3536d4c
Compare
dralley
changed the title
mdellweg
approved these changes
Apr 6, 2021
|
Nice! |
dralley
commented
Apr 6, 2021
| pk__in=publication.repository_version.content | ||
| ).order_by("-pulp_created"): | ||
| for content_artifact in content.contentartifact_set.all(): | ||
| artifact = find_artifact() |
Use a more complex single query rather than N small queries to iterate content artifacts. Using a repositoriy containing 20,000 content units, the publish time is improved from ~23 seconds to 1 second if the content is immediate synced, and 14 seconds (vs. the same) if content is on_demand. closes: #8508 https://pulp.plan.io/issues/8508
dralley
force-pushed
the
publish-improvement
branch
from
3536d4c
to
69182ee
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use a more single query rather than N small queries to iterate content
artifacts.
Using a repositoriy containing 20,000 content units, the publish time
is improved to around 1 second (opposed to ~23 seconds) if the content
is immediate, and 14 seconds (vs. the same) if content is on_demand.
Complexity (in terms of # of queries performed and not taking into account
query complexity) was reduced from 2N + 1 to either a constant or 1N
depending on whether content was synced immediate or on-demand.
Benefit is very noticeable.
closes: #8508
https://pulp.plan.io/issues/8508