Update Celery and Kombu dependencies #4842
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4842 +/- ##
=======================================
Coverage 96.63% 96.63%
=======================================
Files 387 387
Lines 21580 21580
Branches 1223 1223
=======================================
Hits 20854 20854
Misses 611 611
Partials 115 115
Continue to review full report at Codecov.
|
|
In addition to Python 3.6 support, it turns out that this may also be useful for partners running h due to support for new brokers such as SQS in Celery 4.x |
robertknight
force-pushed
the
update-celery
branch
from
dd1afce
to
d608561
Compare
robertknight
force-pushed
the
update-celery
branch
from
d608561
to
30b09c6
Compare
This is required to support Python 3.6. See http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#upgrading-from-celery-3-1 for changes and migration notes.
Update Celery setting names using `celery upgrade settings h/celery.py` as described at http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#lowercase-setting-names
This was an experimental feature that was removed in Celery 4.x. See http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#removed-features
robertknight
force-pushed
the
update-celery
branch
from
30b09c6
to
56c1c13
Compare
| @@ -80,7 +80,7 @@ def devserver(https, web, ws, worker, assets, beat): | |||
| m.add_process('ws', 'gunicorn --name websocket --reload --paste conf/development-websocket.ini %s' % gunicorn_args) | |||
|
|
|||
| if worker: | |||
| m.add_process('worker', 'hypothesis --dev celery worker --autoreload -l INFO') | |||
| m.add_process('worker', 'hypothesis --dev celery worker -l INFO') | |||
There was a problem hiding this comment.
It looks like we don't need this as we have other ways of reloading files automagically when running in dev mode, if I'm understanding what this feature was intended for. (I know it was deprecated but just being thorough about whether we need to replace it with something else or not and I think the answer is no we don't.)
There was a problem hiding this comment.
We have autoreloading of the webpage/API-serving code in h via gunicorn's --reload flag. I don't think we have any replacement mechanism for --autoreload in place here, and I'm not sure it was actually working anyway. I don't know what commonly used alternatives are but I found one suggestion to use watchdog.
I don't think it has been/will be much of an issue in practice because worker code just doesn't get changed that often.
Update Celery from 3.x to current 4.x release.
This is required to support the latest Python 3.x version. For changelog see http://celery.readthedocs.io/en/latest/whatsnew-4.0.html#upgrading-from-celery-3-1 . Note that we were already using 3.1.25 as highlighted in the migration notes.
The only breaking change I encountered was the removal of the
--autoreloadflag. I also applied an automated fix toh/celery.pyto migrate setting names as documented in the changelog.Edit: Elsevier are also interested in this because I believe it should enable them to run h using Amazon SQS (which they already use) instead of setting up a separate RabbitMQ service.