Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PulpImport can overload the task-queue #4068

Closed
ggainey opened this issue Jul 18, 2023 · 4 comments · Fixed by #4160
Closed

PulpImport can overload the task-queue #4068

ggainey opened this issue Jul 18, 2023 · 4 comments · Fixed by #4160
Assignees
@ggainey
Labels
BZ Issue prio-list

Comments

@ggainey
Copy link
Contributor

ggainey commented Jul 18, 2023

Version
main

Describe the bug
PulpImport dispatches a task-per-repository. For large imports, with large repositories, this means that Pulp can't do anything else until the import is finished or nearly so, as all workers are busy importing repositories.

It would be good to teach the import-process to arrange to not use up more than, say, 25% of the available workers.
@ipanova
Copy link
Member

ipanova commented Jul 19, 2023

Also we could introduce a setting for the import task that would make the % value configurable.

I am adding a prio-list to this issue since this problem impairs some of our customer's environment.

@ggainey ggainey self-assigned this Jul 25, 2023
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 26, 2023
@ggainey
Taught import to throttle at 25% of available workers.
d2c5f00 
When we expose  IMPORT_WORKERS_PERCENT, this becomes a configurable percentage.

fixes pulp#4068.
@ggainey ggainey mentioned this issue Jul 26, 2023
Merged
@dralley dralley added the BZ label Jul 27, 2023
@dralley
Copy link
Contributor

dralley commented Jul 27, 2023

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2227093

ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to throttle at 25% of available workers.
c1d8ced 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable.

fixes pulp#4068.
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to throttle at 25% of available workers.
43acefc 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable.

fixes pulp#4068.
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
ccf30a6 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.
ggainey added a commit that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
b5cb1d1 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.
patchback bot pushed a commit that referenced this issue Jul 28, 2023
@ggainey @patchback
Taught import to be able to use less than all-available workers.
9da5315 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
@patchback patchback bot mentioned this issue Jul 28, 2023
Closed
patchback bot pushed a commit that referenced this issue Jul 28, 2023
@ggainey @patchback
Taught import to be able to use less than all-available workers.
d2e967b 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
58f8fcd 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
@ggainey ggainey mentioned this issue Jul 28, 2023
Merged
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
f8d9cd6 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
@ggainey ggainey mentioned this issue Jul 28, 2023
Merged
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
3252194 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
@ggainey ggainey mentioned this issue Jul 28, 2023
Merged
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
ff0c9be 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
ba49746 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
994d485 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
ggainey added a commit to ggainey/pulpcore that referenced this issue Jul 28, 2023
@ggainey
Taught import to be able to use less than all-available workers.
e737ba4 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
@ggainey ggainey mentioned this issue Jul 28, 2023
Merged
dralley pushed a commit that referenced this issue Jul 28, 2023
@ggainey @dralley
Taught import to be able to use less than all-available workers.
711b213 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
dralley pushed a commit that referenced this issue Jul 28, 2023
@ggainey @dralley
Taught import to be able to use less than all-available workers.
eed8222 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
dralley pushed a commit that referenced this issue Jul 28, 2023
@ggainey @dralley
Taught import to be able to use less than all-available workers.
b4e38e4 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
dralley pushed a commit that referenced this issue Jul 28, 2023
@ggainey @dralley
Taught import to be able to use less than all-available workers.
3f96ae1 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
@pulpbot
Copy link
Member

pulpbot commented Jul 31, 2023

https://bugzilla.redhat.com/show_bug.cgi?id=2227908

@pulpbot
Copy link
Member

pulpbot commented Aug 11, 2023

https://bugzilla.redhat.com/show_bug.cgi?id=2231501

ggainey added a commit to ggainey/pulpcore that referenced this issue Aug 21, 2023
@ggainey
Taught import to be able to use less than all-available workers.
4ba4ab0 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes pulp#4068.

(cherry picked from commit b5cb1d1)
@ggainey ggainey mentioned this issue Aug 21, 2023
Merged
dralley pushed a commit that referenced this issue Aug 21, 2023
@ggainey @dralley
Taught import to be able to use less than all-available workers.
b4bae50 
IMPORT_WORKERS_PERCENT is configurable in settings. We will
document/expose this in a future PR, to keep this one maximally
backportable. Default behavior remains "all workers".

fixes #4068.

(cherry picked from commit b5cb1d1)
@evgeni evgeni mentioned this issue Aug 21, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ggainey ggainey

Labels
BZ Issue prio-list
Projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Taught import to be able to use less than all-available workers. ggainey/pulpcore
4 participants
@dralley @ipanova @ggainey @pulpbot