fix: make redis workers to know the current num of workers at startup - #7939
Merged
Conversation
carlosthe19916
force-pushed
the
hotfix/PULP-2149
branch
from
July 31, 2026 06:33
fddedc9 to
de97316
Compare
gerrod3
reviewed
Jul 31, 2026
| @@ -0,0 +1,2 @@ | |||
| Fixed Redis workers polling the database at the single-worker rate during | |||
Contributor
There was a problem hiding this comment.
Can you make this changelog just one line?
Contributor
Author
There was a problem hiding this comment.
Sorry for that, I made that change just now
Signed-off-by: Carlos Feria <2582866+carlosthe19916@users.noreply.github.com>
carlosthe19916
force-pushed
the
hotfix/PULP-2149
branch
from
July 31, 2026 18:09
de97316 to
c109775
Compare
Contributor
Author
|
@gerrod3 Is there anything you would like me to change or reconsider in this particular PR? I admit my limited knowledge of the whole system so I am very open to anything you might suggest :) |
gerrod3
approved these changes
Aug 3, 2026
Backport to 3.115: 💚 backport PR created✅ Backport PR branch: Backported as #7941 🤖 @patchback |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #7912
Addresses: https://redhat.atlassian.net/browse/PULP-2149
Problem
Redis workers find work by repeatedly asking the database for waiting tasks, then sleeping briefly. Sleep is supposed to get longer as the fleet grows, so many workers don’t all pound the DB at once.
But each new worker starts life assuming the fleet has one worker. It only learns the real count on its first heartbeat (~10s later). Until then it polls about every 10 ms.
So a scale-up (e.g. 25 → 150) creates a short window of thousands of SELECTs per second against core_task. The DB saturates, heartbeat writes fail, workers exit, replacements start, and the same aggressive startup polling happens again.
Solution
workers should learn fleet size at startup before first heartbeat
📜 Checklist
See: Pull Request Walkthrough