You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We had a 4.3.1 instance that occasionally crashed on PO import causing the redis queue to go down with it and be lost. This problem has resolved itself after upgrading to 4.4.2. To update all our components we had to re-import PO files for quite a lot of components using weblate loadpo --all --lang <code>. This produced about ~270 celery tasks in the queue. We kept an eye on the log to see whether our earlier problem was resolved. In the log we saw the following deadlock error multiple times.
The error does not actually cause any problems we can find within Weblate, but since it's what looks like a fatal error that was not yet reported on GitHub I wanted to report it for your information.
I already tried
Describe the steps you tried to solve the problem yourself.
I've read and searched the docs and did not find the answer there.
If you didn’t try already, try to search there what you wrote above.
Looked in the GitHub issue queue for related open issues.
To Reproduce the issue
Steps to reproduce the behavior:
Run weblate loadpo --all --lang <langcode> form multiple languages
See the error in
Expected behavior
No errors in the logs
Exception traceback
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/1848] Failure while executing task: OperationalError: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/ForkPoolWorker-5] Failure while executing task: OperationalError: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/1848] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'kwargs', 'traceback', 'exception', 'einfo', 'args', 'task_id'}>, 'sender': <@task: weblate.trans.tasks.perform_load of weblate at 0x7ff99ba17860>, 'task_id': '81b245f9-cb65-4c25-ae7d-b9206a07c307', 'args': [213], 'kwargs': {'force': False, 'langs': ['bg']}, 'traceback': <traceback object at 0x7ff98bd7ff08>, 'einfo': <ExceptionInfo: OperationalError('deadlock detected\nDETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.\nProcess 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (7881,1) in relation "trans_unit"\nSQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"\n')>}
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/ForkPoolWorker-5] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'kwargs', 'traceback', 'exception', 'einfo', 'args', 'task_id'}>, 'sender': <@task: weblate.trans.tasks.perform_load of weblate at 0x7ff99ba17860>, 'task_id': '81b245f9-cb65-4c25-ae7d-b9206a07c307', 'args': [213], 'kwargs': {'force': False, 'langs': ['bg']}, 'traceback': <traceback object at 0x7ff98bd7ff08>, 'einfo': <ExceptionInfo: OperationalError('deadlock detected\nDETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.\nProcess 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (7881,1) in relation "trans_unit"\nSQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"\n')>}
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/1848] Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
celery-main stderr |
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr |
celery-main stderr | Traceback (most recent call last):
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 405, in trace_task
celery-main stderr | R = retval = fun(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 697, in __protected_call__
celery-main stderr | return self.run(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/autoretry.py", line 34, in run
celery-main stderr | return task._orig_run(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 86, in perform_load
celery-main stderr | force=force, langs=langs, changed_template=changed_template, from_link=from_link
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1751, in create_translations
celery-main stderr | force, langs, request, changed_template, from_link
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1875, in _create_translations
celery-main stderr | self.progress_step()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/transaction.py", line 232, in __exit__
celery-main stderr | connection.commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr | return func(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 266, in commit
celery-main stderr | self._commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr | raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | django.db.utils.OperationalError: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
celery-main stderr | [2021-01-27 10:56:50,426: ERROR/ForkPoolWorker-5] Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
celery-main stderr |
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr |
celery-main stderr | Traceback (most recent call last):
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 405, in trace_task
celery-main stderr | R = retval = fun(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 697, in __protected_call__
celery-main stderr | return self.run(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/autoretry.py", line 34, in run
celery-main stderr | return task._orig_run(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 86, in perform_load
celery-main stderr | force=force, langs=langs, changed_template=changed_template, from_link=from_link
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1751, in create_translations
celery-main stderr | force, langs, request, changed_template, from_link
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1875, in _create_translations
celery-main stderr | self.progress_step()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/transaction.py", line 232, in __exit__
celery-main stderr | connection.commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr | return func(*args, **kwargs)
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 266, in commit
celery-main stderr | self._commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr | raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 242, in _commit
celery-main stderr | return self.connection.commit()
celery-main stderr | django.db.utils.OperationalError: deadlock detected
celery-main stderr | DETAIL: Process 9532 waits for ShareLock on transaction 2185973; blocked by process 13265.
celery-main stderr | Process 13265 waits for ShareLock on transaction 2183699; blocked by process 9532.
celery-main stderr | HINT: See server log for query details.
celery-main stderr | CONTEXT: while locking tuple (7881,1) in relation "trans_unit"
celery-main stderr | SQL statement "SELECT 1 FROM ONLY "public"."trans_unit" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"
celery-main stderr |
Other instances of this error showed different process numbers, forkpoolworker number and tuple (x,y) values but were otherwise identical.
System check identified some issues:
INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
HINT: https://docs.weblate.org/en/weblate-4.4.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
HINT: https://docs.weblate.org/en/weblate-4.4.2/admin/backup.html
System check identified 2 issues (1 silenced).
The text was updated successfully, but these errors were encountered:
I've already seen this with one customer, but unfortunately, we didn't find the cause there yet...
This can probably sometimes happen when multiple updates are running in parallel, and your description seems to match that. I still wonder why we've never seen this on our production servers...
The proper long-term solution for this is to avoid holding key lock for updates, but that will be possible after Django 3.2 is released, see #5227
Glad to hear it's a known issue :) I'm not entirely sure what it's locking on. Could it have something to do with components that share a repository (maybe they also share other things)? We have quite a few of those.
No, it's lock on the units table. Weblate holds it to avoid concurrent updates of the units, but in current Django implementation it doesn't hold only row level lock, but also primary key lock. And that blocks insertion into that table for that time.
Describe the issue
We had a 4.3.1 instance that occasionally crashed on PO import causing the redis queue to go down with it and be lost. This problem has resolved itself after upgrading to 4.4.2. To update all our components we had to re-import PO files for quite a lot of components using
weblate loadpo --all --lang <code>
. This produced about ~270 celery tasks in the queue. We kept an eye on the log to see whether our earlier problem was resolved. In the log we saw the following deadlock error multiple times.The error does not actually cause any problems we can find within Weblate, but since it's what looks like a fatal error that was not yet reported on GitHub I wanted to report it for your information.
I already tried
Describe the steps you tried to solve the problem yourself.
If you didn’t try already, try to search there what you wrote above.
To Reproduce the issue
Steps to reproduce the behavior:
weblate loadpo --all --lang <langcode>
form multiple languagesExpected behavior
No errors in the logs
Exception traceback
Other instances of this error showed different process numbers, forkpoolworker number and
tuple (x,y)
values but were otherwise identical.Server configuration and status
Weblate installation: Docker
Weblate deploy checks
The text was updated successfully, but these errors were encountered: