Buggy behaviour after upgrading from 4.0.4 to 4.2.2 (errors when doing check --deploy and performance report)

[erciccione]

Describe the issue

After upgrading from 4.0.4 to 4.2.2 i have some problems:

I'm experiencing a weird behaviour when trying to add a new language to a component. Doing it with the UI fails giving a server error, but i see a new file being added in the git repository (only added physically, but not committed or tracked). If i commit the new language file manually, the UI doesn't seem to notice the difference. I don't even see the commit in the "repository details".

This only happens for a new component added after the upgrade. Adding a new language to an old component works fine.

Also, administration -> performance report gives a server error. The error given is argument of type 'NoneType' is not iterable

check --deploy fails:

  File "/var/www/weblate-env/bin/weblate", line 10, in <module>
    sys.exit(main())
  File "/var/www/weblate-env/lib/python3.6/site-packages/weblate/runner.py", line 32, in main
    execute_from_command_line(argv)
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/commands/check.py", line 69, in handle
    databases=options['databases'],
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/management/base.py", line 396, in check
    databases=databases,
  File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/checks/registry.py", line 70, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/var/www/weblate-env/lib/python3.6/site-packages/weblate/utils/checks.py", line 212, in check_celery
    for key, value in current.items()
  File "/var/www/weblate-env/lib/python3.6/site-packages/weblate/utils/checks.py", line 213, in <listcomp>
    if key not in pong or value != pong[key]
TypeError: argument of type 'NoneType' is not iterable

check succeedes but with weird git-related warnings.

Server configuration and status

Weblate deploy checks

 * Weblate: 4.2.2
 * Django: 3.1.2
 * siphashc: 2.1
 * Whoosh: 2.7.4
 * translate-toolkit: 3.1.1
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.2
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 50.3.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.5
 * celery: 4.4.7
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.19.1
 * pygobject: 3.38.0
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.9
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.6.7
 * Git: 2.17.1
 * psycopg2-binary: 2.8.3
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.12
 * tesserocr: 2.5.1
 * boto3: 1.15.10
 * zeep: 3.4.0
 * aeidon: 1.7.0
 * Redis server: 4.0.9
 * PostgreSQL server: 10.12
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 4.15.0-12-generic (x86_64)

[github-actions]

This issue looks like a support question. We try to answer these reasonably fast, but in case you are looking for faster resolution, please consider purchasing support subscription and make Weblate stronger.

[nijel]

You have probably not restarted the Celery process and it still runs old code.

[erciccione]

I already restarted the celery process (i use the script in examples), which seems to be working fine. I just double checked the settings.py file and the content of the celery script. They are both updated, but the problems persist.

Note that the script starts correctly and celery_queues works and shows nothing is queued.

[erciccione]

I restarted the celery script again and i'm noticing that adding a new language now works as expected, so that part is solved. I'm still experiencing a server error when clicking on "performance report" and deleting a component doesn't seem to work, even if o do get the Translation component was scheduled for removal. message (the celery queue remains empty).

check --deploy still fails

[nijel]

Can you check celery logs for errors? Maybe the script in examples didn't probably kill all processes as some queues were removed in recent releases. Try killing all celery processes manually and start it again.

[erciccione]

yes, many celery processes were still there, killing them fixed check --deploy. The logs show me an error ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.

[erciccione]

Yes, celery queues seem to be stuck, check --deploy : (weblate.E019) The Celery does not process tasks, or is too slow in processing them.

From the logs:

[2020-10-02 11:31:53,166: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Agent, Tasks, Control, Heart, event loop}
[2020-10-02 11:31:53,170: DEBUG/MainProcess] | Worker: Starting Beat
[2020-10-02 11:31:53,172: DEBUG/MainProcess] ^-- substep ok
[2020-10-02 11:31:53,172: DEBUG/MainProcess] | Worker: Starting Hub
[2020-10-02 11:31:53,172: DEBUG/MainProcess] ^-- substep ok
[2020-10-02 11:31:53,172: DEBUG/MainProcess] | Worker: Starting Pool
[2020-10-02 11:31:53,206: INFO/Beat] beat: Starting...
[2020-10-02 11:31:53,209: ERROR/Beat] Removing corrupted schedule file 'celerybeat-schedule': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
  File "/var/www/weblate-env/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/www/weblate-env/lib/python3.6/site-packages/celery/beat.py", line 519, in setup_schedule
    self._store = self._open_schedule()
  File "/var/www/weblate-env/lib/python3.6/site-packages/celery/beat.py", line 509, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.6/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.6/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.6/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
[2020-10-02 11:31:53,216: DEBUG/Beat] Current schedule:
<ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
[2020-10-02 11:31:53,216: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes
[2020-10-02 11:31:53,216: DEBUG/Beat] beat: Waking up in 5.00 minutes.
[2020-10-02 11:31:53,423: DEBUG/MainProcess] ^-- substep ok
[2020-10-02 11:31:53,424: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-10-02 11:31:53,424: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-10-02 11:31:53,476: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds... (1/100)

Looking it up

[erciccione]

The problem was caused by the fact that somehow pip installed celery 5.0, which is incompatible with Weblate. I downgraded celery to 4.4.5 and all problems seem to be solved. 17b9d95 should avoid the problem for the future.

Thank you for your help @nijel

[github-actions]

The issue you have reported seems to be resolved now.

• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[nijel]

Ah, there is issue with the way the example starts Celery, it is broken in 4.4.7 and newer, see celery/celery#6285. Workaround for this is in eb419be