Skip to content

fix: make redis workers to know the current num of workers at startup - #7939

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
carlosthe19916:hotfix/PULP-2149
Aug 3, 2026
Merged

fix: make redis workers to know the current num of workers at startup#7939
gerrod3 merged 1 commit into
pulp:mainfrom
carlosthe19916:hotfix/PULP-2149

Conversation

@carlosthe19916

@carlosthe19916 carlosthe19916 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Fixes: #7912
Addresses: https://redhat.atlassian.net/browse/PULP-2149

Problem

Redis workers find work by repeatedly asking the database for waiting tasks, then sleeping briefly. Sleep is supposed to get longer as the fleet grows, so many workers don’t all pound the DB at once.

But each new worker starts life assuming the fleet has one worker. It only learns the real count on its first heartbeat (~10s later). Until then it polls about every 10 ms.

So a scale-up (e.g. 25 → 150) creates a short window of thousands of SELECTs per second against core_task. The DB saturates, heartbeat writes fail, workers exit, replacements start, and the same aggressive startup polling happens again.

Solution

workers should learn fleet size at startup before first heartbeat

📜 Checklist

  • Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
  • A changelog entry or entries has been added for any significant changes
  • Follows the Pulp policy on AI Usage
  • (For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

@@ -0,0 +1,2 @@
Fixed Redis workers polling the database at the single-worker rate during

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this changelog just one line?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for that, I made that change just now

Signed-off-by: Carlos Feria <2582866+carlosthe19916@users.noreply.github.com>
@carlosthe19916

Copy link
Copy Markdown
Contributor Author

@gerrod3 Is there anything you would like me to change or reconsider in this particular PR? I admit my limited knowledge of the whole system so I am very open to anything you might suggest :)

@gerrod3
gerrod3 merged commit d212f0b into pulp:main Aug 3, 2026
13 of 14 checks passed
@patchback

patchback Bot commented Aug 3, 2026

Copy link
Copy Markdown

Backport to 3.115: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.115/d212f0b5737f578fef0aaedb5e799872bca7202e/pr-7939

Backported as #7941

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RedisWorker should query fleet size at startup before first heartbeat

2 participants