Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split the import_collection task into two tasks to reduce the amount of time the repository lock is held #709

Closed
pulpbot opened this issue Nov 17, 2021 · 2 comments

Comments

@pulpbot
Copy link
Member

pulpbot commented Nov 17, 2021

Author: @bmbouter (bmbouter)

Redmine Issue: 6677, https://pulp.plan.io/issues/6677


Background

The import_collection task both analyzes a collection here and then creates content and adds it to a repository version here.

Problem

The import_collection runs long enough that to import a lot of content into a repository serializes these long tasks making the whole workload take long. It takes longer than it should because the collection analysis portion of the task runtime takes the majority of the time and the lock on repository is not needed during that. The lock is only needed when creating the RepositoryVersion itself.

Solution

  1. Split import_collection into two tasks. analyze_and_create_collection and then add_collection_to_repository.
  2. Have the analyze_and_create_collection task dispatch the add_collection_to_repository. The analyze_and_create_collection task requires no lock. The add_collection_to_respository should have the lock on the repository.
@pulpbot
Copy link
Member Author

pulpbot commented Nov 17, 2021

From: pulpbot (pulpbot)
Date: 2020-05-19T18:07:55Z


PR: #314

@gerrod3
Copy link
Contributor

gerrod3 commented Jul 12, 2022

Duplicate of #1105. Closing this one and using new one to track work.

@gerrod3 gerrod3 closed this as completed Jul 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants