Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Update runservice worker scripts to disable unneeded features #1371

Merged
merged 1 commit into from Nov 14, 2023

Conversation

baumandm
Copy link
Contributor

We've encountered issues where workers would stop receiving tasks after an error message like this:

consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)

I was able to reproduce this locally via Docker by manually bouncing the Redis container.

After some research, we came across suggestions to disable the heartbeat/gossip/mingle features of Celery. In my local testing it appears this change eliminates (or at least reduces) the issue where the workers cannot reconnect to Redis after a connection failure.

We have been running this in production for over a month and everything works as expected, including Flower.

@czgu czgu merged commit 09ffc40 into pinterest:master Nov 14, 2023
2 checks passed
aidenprice pushed a commit to arrowtail-precision/querybook that referenced this pull request Jan 3, 2024
@baumandm baumandm deleted the external/worker-redis branch January 10, 2024 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants