-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve publish speed by between 2x and 20x #493
Conversation
28d24d8
to
960b818
Compare
Attached issue: https://pulp.plan.io/issues/8508 |
13c1040
to
3536d4c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
If i read it correctly, it scales accordingly for mixed repo versions: 1 + #(on-demand artifacts)
Nice! |
pk__in=publication.repository_version.content | ||
).order_by("-pulp_created"): | ||
for content_artifact in content.contentartifact_set.all(): | ||
artifact = find_artifact() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2N + 1 queries
Use a more complex single query rather than N small queries to iterate content artifacts. Using a repositoriy containing 20,000 content units, the publish time is improved from ~23 seconds to 1 second if the content is immediate synced, and 14 seconds (vs. the same) if content is on_demand. closes: #8508 https://pulp.plan.io/issues/8508
3536d4c
to
69182ee
Compare
Use a more single query rather than N small queries to iterate content
artifacts.
Using a repositoriy containing 20,000 content units, the publish time
is improved to around 1 second (opposed to ~23 seconds) if the content
is immediate, and 14 seconds (vs. the same) if content is on_demand.
Complexity (in terms of # of queries performed and not taking into account
query complexity) was reduced from 2N + 1 to either a constant or 1N
depending on whether content was synced immediate or on-demand.
Benefit is very noticeable.
closes: #8508
https://pulp.plan.io/issues/8508