You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While working on #3566, discovered that the scheduled tasks and cron jobs stopped running in dev and stage environments. The celery worker and beat apps on dev and stage environments failed to start during circle builds and also manual restart of those apps did work either. The status of these two apps remained in CRASHED status and non of these two app instances were UP and RUNNING.
After taking a close look at the celery worker logs, it appeared that the kombu AMQP(Advanced Messaging Queue Protocol) is expecting a higher version of redis than what we have currently in openFEC/requirements.txt.
Here is the log stack trace from celery worker app: 2019-03-06T14:01:56.64-0500 [APP/PROC/WEB/0] ERR kombu.exceptions.VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6
Our development stack has redis 2.10.6 which is not compatible with the latest kombu4.4.0. This caused the celery worker/beat apps CRASH in dev and stage environments.
Fix/Resolution:
Celery worker/beat app were UP and RUNNING after upgrading redis version to 3.2.0 and deploy the changes to dev and stage spaces.
Redis upgrade work is implemented in this PR #3605
The text was updated successfully, but these errors were encountered:
While working on #3566, discovered that the scheduled tasks and cron jobs stopped running in
dev
andstage
environments. The celery worker and beat apps ondev
andstage
environments failed to start during circle builds and also manual restart of those apps did work either. The status of these two apps remained inCRASHED
status andnon
of these two app instances wereUP
andRUNNING
.After taking a close look at the celery worker logs, it appeared that the
kombu
AMQP(Advanced Messaging Queue Protocol) is expecting a higher version ofredis
than what we have currently inopenFEC/requirements.txt
.Here is the log stack trace from celery worker app:
2019-03-06T14:01:56.64-0500 [APP/PROC/WEB/0] ERR kombu.exceptions.VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6
Problem:
Sprint8.2 release cut happened on
Mar 5, 2019
. During the build,celery4.1.1
grabbed the newer version ofkombu >=4.2.0,<5.0
. The kombu package was updated onMar 3, 2019
and we are not aware of this change. The newer version ofkombu4.4.0
is compatible withredis3.2.0
only.Kombu :https://github.com/celery/celery/blob/master/requirements/default.txt
Redis :https://github.com/celery/celery/blob/master/requirements/extras/redis.txt
Our development stack has redis 2.10.6 which is not compatible with the latest kombu4.4.0. This caused the celery worker/beat apps CRASH in
dev
andstage
environments.Fix/Resolution:
Celery worker/beat app were UP and RUNNING after upgrading redis version to 3.2.0 and deploy the changes to
dev
andstage
spaces.Redis upgrade work is implemented in this PR #3605
The text was updated successfully, but these errors were encountered: