Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update redis to fix celery worker/beat connectivity issue when kombu is upgraded #3607

Closed
pkfec opened this issue Mar 7, 2019 · 1 comment
Assignees
Milestone

Comments

@pkfec
Copy link
Contributor

pkfec commented Mar 7, 2019

While working on #3566, discovered that the scheduled tasks and cron jobs stopped running in dev and stage environments. The celery worker and beat apps on dev and stage environments failed to start during circle builds and also manual restart of those apps did work either. The status of these two apps remained in CRASHED status and non of these two app instances were UP and RUNNING.

After taking a close look at the celery worker logs, it appeared that the kombu AMQP(Advanced Messaging Queue Protocol) is expecting a higher version of redis than what we have currently in openFEC/requirements.txt.

Here is the log stack trace from celery worker app:
2019-03-06T14:01:56.64-0500 [APP/PROC/WEB/0] ERR kombu.exceptions.VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6

Problem:
Sprint8.2 release cut happened on Mar 5, 2019. During the build, celery4.1.1 grabbed the newer version of kombu >=4.2.0,<5.0. The kombu package was updated on Mar 3, 2019 and we are not aware of this change. The newer version of kombu4.4.0 is compatible with redis3.2.0 only.
Kombu :https://github.com/celery/celery/blob/master/requirements/default.txt
Redis :https://github.com/celery/celery/blob/master/requirements/extras/redis.txt

Our development stack has redis 2.10.6 which is not compatible with the latest kombu4.4.0. This caused the celery worker/beat apps CRASH in dev and stage environments.

Fix/Resolution:
Celery worker/beat app were UP and RUNNING after upgrading redis version to 3.2.0 and deploy the changes to dev and stage spaces.

Redis upgrade work is implemented in this PR #3605

@pkfec pkfec self-assigned this Mar 7, 2019
@pkfec pkfec added this to the Sprint 8.3 milestone Mar 8, 2019
@pkfec
Copy link
Contributor Author

pkfec commented Mar 8, 2019

The fix is deployed to dev and stage spaces already. Fix will be deployed toprod during the release-20190312.

Here is the PR to track the changes : #3605

Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant