Celery does not parse correctly weblate configuration

[GovanifY]

Describe the bug

When trying to setup weblate I have ran into an issue where celery is defaulting to its amqp settings instead of the redis one I configured in settings.py. The correct settings shows up in list_versions too, leading to an unusable weblate install. Reproducible with the default settings included in the documentation; celery started from ./weblate/examples/celery start .
To Reproduce

Install weblate through virtualenv with postgresql, configure it as you would. The issue is reproducible with any webserver, be it runserver or gunicorn.

Expected behavior

It would be neat if celery actually processed my tasks

Server configuration and status

(weblate-env) weblate@fujiwara ~ $ weblate list_versions
 * Weblate: 4.2.2
 * Django: 3.1.1
 * siphashc: 2.1
 * Whoosh: 2.7.4
 * translate-toolkit: 3.0.0
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.1
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 50.2.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.5
 * celery: 4.4.7
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.19.1
 * pygobject: 3.36.1
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.8
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.6.10
 * Git: 2.26.2
 * psycopg2-binary: 2.8.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.12
 * boto3: 1.14.56
 * zeep: 3.4.0
 * aeidon: 1.7.0
 * Mercurial: 5.2.2
 * git-svn: 2.26.2
 * git-review: 1.28.0
 * hub: 2.14.2
 * lab: 0.16
 * Redis server: 5.0.8
 * PostgreSQL server: 12.2
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 5.4.38-gentoo (x86_64)

Weblate deploy checks

(weblate-env) weblate@fujiwara ~ $ weblate check --deploy
SystemCheckError: System check identified some issues:

CRITICALS:
?: (weblate.E019) The Celery does not process tasks, or is too slow in processing them.
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#celery

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.xwiki-fullpage) Failure in loading handler for xwiki-fullpage file format: module 'translate.storage.properties' has no attribute 'XWikiFullPage'
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.xwiki-java-properties) Failure in loading handler for xwiki-java-properties file format: module 'translate.storage.properties' has no attribute 'xwikifile'
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.xwiki-page-properties) Failure in loading handler for xwiki-page-properties file format: module 'translate.storage.properties' has no attribute 'XWikiPageProperties'
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/backup.html

System check identified 13 issues (1 silenced).

** Relevant system logs **

[2020-09-05 12:02:53,905: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds... (1/100)

[2020-09-05 12:02:55,920: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds... (2/100)

[2020-09-05 12:02:59,942: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 6.00 seconds... (3/100)

[2020-09-05 12:03:05,963: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 8.00 seconds... (4/100)

[2020-09-05 12:03:13,988: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 10.00 seconds... (5/100)

[GovanifY]

I also am not the only one experiencing this issue as this seems related to #4261 , altho with a simpler setup

[GovanifY]

Downgrading to celery==4.4.0 fixed this issue, celery have deprecated their BROKER_URL uppercase for their lowercase settings and are under testing in the 5.0 rc release, we should probably update the requirements.txt to ask for at least 4.4.7 and update the settings to work with the newer versions

[nijel]

The current dependency is:

celery[redis]>=4.4.5,<4.5

So Celery 5.0 is clearly not supported. Why did you install 5.0rc?

[GovanifY]

@nijel I did not, if you read the check_version you'd see I have installed 4.4.7, which already deprecates it! I followed the virtualenv setup on your documentation, and it installs it by default

[nijel]

Weblate works just fine with 4.4.7, that's what we run on all our cloud hosting instances (for example see https://hosted.weblate.org/about/).

Downgrading to celery==4.4.0 fixed this issue,

Can you please try which is the first version that breaks things?

celery have deprecated their BROKER_URL uppercase for their lowercase settings

The Celery settings in Django are still uppercase, see https://docs.celeryproject.org/en/master/django/first-steps-with-django.html#using-celery-with-django

You're second one to run into this, so apparently something is documented wrongly or some code is wrong and I'd like to understand what is that.

[GovanifY]

Can you please try which is the first version that breaks things?

4.4.6 works, it got deprecated as of 4.4.7 it seems

[nijel]

What got deprecated? I've already talked about settings which is not deprecated in way we use them. I can see no depreciation added between these two releases: celery/celery@v4.4.6...v4.4.7

[GovanifY]

@nijel not sure, I can repro it this way: https://asciinema.org/a/uGpLeAUMbMgm4gT0TSdMCRZNe

[nijel]

The root cause seems to be celery/celery#6285 - the celery multi which is used in the example script discards some args when executing workers. I will add workaround to the example script as that is the only documented way to start Celery workers which is affected by this.

It works on our production sites because we don't use celery multi but we run it as a system service (see https://docs.weblate.org/en/latest/admin/install.html#running-celery-as-system-service). Similar setup is used in our Docker containers, so these are not affected as well.

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.