Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can not remove lots of components by sending requests to API in a short time period due to deadlock #4415

Closed
exzhawk opened this issue Aug 31, 2020 · 1 comment
Closed

Can not remove lots of components by sending requests to API in a short time period due to deadlock #4415

exzhawk opened this issue Aug 31, 2020 · 1 comment
Assignees
@nijel
Labels
bug Something is broken.
Milestone
4.2.2

Comments

@exzhawk
Copy link
Contributor

exzhawk commented Aug 31, 2020

Describe the bug

When removing lots of components using DELETE /api/components/(string:project)/(string:component)/. Deadlocks may raise and not all components will get removed.

A few hours later after DELETE requests, logs stopped with no more CPU consumption. But only several of them were removed, leaving logs below.

To Reproduce

This happened after #4398

After removing components failed in the addon script, I tried to remove them using API.
I list all the unwanted components and sent a bunch of DELETE requests to the server.

Expected behavior

All unwanted components are removed eventually.

Screenshots

Server configuration and status

Exception traceback

weblate_1   | celery-main stderr | [2020-08-29 04:26:52,377: INFO/MainProcess] Received task: weblate.trans.tasks.component_removal[96b2ff92-8ef2-4da5-90be-0dc156759223]  
weblate_1   | celery-main stderr | ERROR Failure while executing task: OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | [2020-08-29 04:27:05,722: ERROR/ForkPoolWorker-2269] Failure while executing task: OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'exception', 'kwargs', 'task_id', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_removal of weblate at 0x7f9d9feae8d0>, 'task_id': 'be2f6bb8-94ab-4f4e-9e66-a76b7be38c12', 'args': [5821, 3], 'kwargs': {}, 'traceback': <traceback object at 0x7f9d822f5588>, 'einfo': <ExceptionInfo: OperationalError('deadlock detected\nDETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.\nProcess 24836 waits for ShareLock on transaction 105389; blocked by process 24833.\nHINT:  See server log for query details.\nCONTEXT:  while updating tuple (18,112) in relation "trans_alert"\n')>}
weblate_1   | celery-main stderr | [2020-08-29 04:27:05,722: ERROR/ForkPoolWorker-2269] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'exception', 'kwargs', 'task_id', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_removal of weblate at 0x7f9d9feae8d0>, 'task_id': 'be2f6bb8-94ab-4f4e-9e66-a76b7be38c12', 'args': [5821, 3], 'kwargs': {}, 'traceback': <traceback object at 0x7f9d822f5588>, 'einfo': <ExceptionInfo: OperationalError('deadlock detected\nDETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.\nProcess 24836 waits for ShareLock on transaction 105389; blocked by process 24833.\nHINT:  See server log for query details.\nCONTEXT:  while updating tuple (18,112) in relation "trans_alert"\n')>}
weblate_1   | celery-main stderr | ERROR Failure while executing task
weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | The above exception was the direct cause of the following exception:
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
weblate_1   | celery-main stderr |     R = retval = fun(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
weblate_1   | celery-main stderr |     return self.run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 171, in _inner
weblate_1   | celery-main stderr |     reraise(*exc_info)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/_compat.py", line 57, in reraise
weblate_1   | celery-main stderr |     raise value
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 166, in _inner
weblate_1   | celery-main stderr |     return f(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 299, in component_removal
weblate_1   | celery-main stderr |     obj.delete()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/utils/db.py", line 127, in delete
weblate_1   | celery-main stderr |     return collector.delete()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/deletion.py", line 436, in delete
weblate_1   | celery-main stderr |     sender=model, instance=obj, using=self.using
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in send
weblate_1   | celery-main stderr |     for receiver in self._live_receivers(sender)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in <listcomp>
weblate_1   | celery-main stderr |     for receiver in self._live_receivers(sender)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/utils/decorators.py", line 30, in wrapper
weblate_1   | celery-main stderr |     signal_handler(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/__init__.py", line 223, in post_delete_linked
weblate_1   | celery-main stderr |     instance.linked_component.update_alerts()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2313, in update_alerts
weblate_1   | celery-main stderr |     self.add_alert("DuplicateFilemask", duplicates=duplicates)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1629, in add_alert
weblate_1   | celery-main stderr |     component.add_alert(alert, **details)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1626, in add_alert
weblate_1   | celery-main stderr |     obj.save()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/alert.py", line 61, in save
weblate_1   | celery-main stderr |     super().save(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 751, in save
weblate_1   | celery-main stderr |     force_update=force_update, update_fields=update_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 789, in save_base
weblate_1   | celery-main stderr |     force_update, using, update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 870, in _save_table
weblate_1   | celery-main stderr |     forced_update)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 923, in _do_update
weblate_1   | celery-main stderr |     return filtered._update(values) > 0
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 803, in _update
weblate_1   | celery-main stderr |     return query.get_compiler(self.db).execute_sql(CURSOR)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1515, in execute_sql
weblate_1   | celery-main stderr |     cursor = super().execute_sql(result_type)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
weblate_1   | celery-main stderr |     cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute
weblate_1   | celery-main stderr |     return real_execute(self, sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute
weblate_1   | celery-main stderr |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
weblate_1   | celery-main stderr |     return executor(sql, params, many, context)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
weblate_1   | celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | django.db.utils.OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | [2020-08-29 04:27:05,723: ERROR/ForkPoolWorker-2269] Failure while executing task
weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | The above exception was the direct cause of the following exception:
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
weblate_1   | celery-main stderr |     R = retval = fun(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
weblate_1   | celery-main stderr |     return self.run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 171, in _inner
weblate_1   | celery-main stderr |     reraise(*exc_info)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/_compat.py", line 57, in reraise
weblate_1   | celery-main stderr |     raise value
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 166, in _inner
weblate_1   | celery-main stderr |     return f(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 299, in component_removal
weblate_1   | celery-main stderr |     obj.delete()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/utils/db.py", line 127, in delete
weblate_1   | celery-main stderr |     return collector.delete()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/deletion.py", line 436, in delete
weblate_1   | celery-main stderr |     sender=model, instance=obj, using=self.using
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in send
weblate_1   | celery-main stderr |     for receiver in self._live_receivers(sender)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in <listcomp>
weblate_1   | celery-main stderr |     for receiver in self._live_receivers(sender)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/utils/decorators.py", line 30, in wrapper
weblate_1   | celery-main stderr |     signal_handler(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/__init__.py", line 223, in post_delete_linked
weblate_1   | celery-main stderr |     instance.linked_component.update_alerts()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2313, in update_alerts
weblate_1   | celery-main stderr |     self.add_alert("DuplicateFilemask", duplicates=duplicates)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1629, in add_alert
weblate_1   | celery-main stderr |     component.add_alert(alert, **details)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1626, in add_alert
weblate_1   | celery-main stderr |     obj.save()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/alert.py", line 61, in save
weblate_1   | celery-main stderr |     super().save(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 751, in save
weblate_1   | celery-main stderr |     force_update=force_update, update_fields=update_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 789, in save_base
weblate_1   | celery-main stderr |     force_update, using, update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 870, in _save_table
weblate_1   | celery-main stderr |     forced_update)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 923, in _do_update
weblate_1   | celery-main stderr |     return filtered._update(values) > 0
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 803, in _update
weblate_1   | celery-main stderr |     return query.get_compiler(self.db).execute_sql(CURSOR)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1515, in execute_sql
weblate_1   | celery-main stderr |     cursor = super().execute_sql(result_type)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
weblate_1   | celery-main stderr |     cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute
weblate_1   | celery-main stderr |     return real_execute(self, sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute
weblate_1   | celery-main stderr |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
weblate_1   | celery-main stderr |     return executor(sql, params, many, context)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
weblate_1   | celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | django.db.utils.OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,112) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-notify stderr | [2020-08-29 04:27:06,718: INFO/MainProcess] Received task: weblate.accounts.tasks.notify_change[1b04ef9b-e548-4b08-9dee-04cd869c86a9]  
weblate_1   | celery-main stderr | [2020-08-29 04:27:08,421: INFO/MainProcess] Received task: weblate.trans.tasks.component_removal[5527a8f2-4af9-45b7-bae7-37ba4d32b7c9]  
weblate_1   | celery-main stderr | ERROR Failure while executing task: OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24836 waits for ShareLock on transaction 105393; blocked by process 24839.
weblate_1   | celery-main stderr | Process 24839 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,113) in relation "trans_alert"
weblate_1   | celery-main stderr | 
weblate_1   | celery-main stderr | [2020-08-29 04:27:24,429: ERROR/ForkPoolWorker-2270] Failure while executing task: OperationalError: deadlock detected
weblate_1   | celery-main stderr | DETAIL:  Process 24836 waits for ShareLock on transaction 105393; blocked by process 24839.
weblate_1   | celery-main stderr | Process 24839 waits for ShareLock on transaction 105391; blocked by process 24836.
weblate_1   | celery-main stderr | HINT:  See server log for query details.
weblate_1   | celery-main stderr | CONTEXT:  while updating tuple (18,113) in relation "trans_alert"
weblate_1   | celery-main stderr |

Additional context

Since there is a deadlock, it might be done one by one. I rewrote the script, making it always checking if the last component is removed before sending DELETE request for the next component. This scripts worked and all unwanted components are removed.
@nijel nijel added the bug Something is broken. label Sep 1, 2020
@nijel nijel self-assigned this Sep 1, 2020
@nijel nijel added this to the 4.2.2 milestone Sep 1, 2020
nijel added a commit that referenced this issue Sep 1, 2020
@nijel
Component: Avoid updating existing alerts when removing
5b6fd51 
In case of removing many components this can lead to deadlock on the
database side as all removed linked components are trying to update same
alerts.

Fixe #4415
@nijel nijel closed this as completed Sep 1, 2020
@github-actions
Copy link

github-actions bot commented Sep 1, 2020

Thank you for your report, the issue you have reported has just been fixed.

  • In case you see a problem with the fix, please comment on this issue.
  • In case you see a similar problem, please open a separate issue.
  • If you are happy with the outcome, consider supporting Weblate by donating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nijel nijel

Labels
bug Something is broken.
Projects
None yet
Milestone
4.2.2
Development

No branches or pull requests
2 participants
@nijel @exzhawk