Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory #4583

Closed
bilalcorbacioglu opened this issue Sep 24, 2020 · 11 comments
Closed

Out of memory #4583

bilalcorbacioglu opened this issue Sep 24, 2020 · 11 comments
Assignees
Labels
bug Something is broken.
Milestone

Comments

@bilalcorbacioglu
Copy link

Describe the bug

Hello,

We've been getting an "Out of Memory" error periodically since we update to Weblate 4.2.2. Especially when we edit some components, For example;

Source: And that is the end of our story.
Target: Voilà, c'est la fin de l'histoire. -> Voilà, c’est la fin de l’histoire.

Source: Touch a button!
Target: Touche un bouton ! -> Touche un bouton/u202f!

  • I'm using a "Customized rule" but when I deactivate it, the error(Out of memory) continues.
  • No records are forwarded to the error file during/after the error.

Screenshots

Before click "Save"
Screen Shot 2020-09-24 at 09 47 06
After Click "Save"
Screen Shot 2020-09-24 at 09 47 45

Server configuration and status

 * Weblate: 4.2.2
 * Django: 3.1.1
 * siphashc: 1.3
 * Whoosh: 2.7.4
 * translate-toolkit: 3.0.0
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.1
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.5
 * celery: 4.4.7
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.7
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.8.5
 * psycopg2-binary: 2.8.5
 * phply: 1.2.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.10
 * tesserocr: 2.5.1
 * akismet: 1.1
 * boto3: 1.14.53
 * zeep: 3.4.0
 * aeidon: 1.7.0
 * iniparse: 0.5
 * mysqlclient: 2.0.1
 * Mercurial: 5.5.1
 * git-svn: 2.20.1
 * git-review: 1.28.0
 * hub: 2.13.0
 * lab: 0.16
 * Redis server: 5.0.9
 * PostgreSQL server: 11.7
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.gmail.com
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-117-generic (x86_64)
@nijel
Copy link
Member

nijel commented Sep 24, 2020

You might want to reduce number of processes running inside the container, see https://docs.weblate.org/en/latest/admin/install/docker.html#container-settings

@nijel nijel added the question This is more a question for the support than an issue. label Sep 24, 2020
@github-actions
Copy link

This issue looks like a support question. We try to answer these reasonably fast, but in case you are looking for faster resolution, please consider purchasing support subscription and make Weblate stronger.

@bilalcorbacioglu
Copy link
Author

bilalcorbacioglu commented Sep 25, 2020

This method did a little help with Weblate's freezing problem. However, Weblate continues to freezes after a few requests because CPU usage is always high(After trying to translate the translation). I also share a few other error logs I can catch below.

weblate_1   | celery-main stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,481: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[f6356314-2794-4a9f-a3e6-a9d3e26b37d1]  
weblate_1   | celery-backup stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:53,270: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-main stderr | 45 seconds.  [orig: 2020-09-25 06:36:46.970975 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-main stderr | 
weblate_1   | celery-notify stderr | [2020-09-25 06:36:46,899: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-notify stderr | 44 seconds.  [orig: 2020-09-25 06:36:45.495046 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-notify stderr | 
weblate_1   | celery-translate stderr | [2020-09-25 06:36:46,899: WARNING/MainProcess] Substantial drift from notify@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-translate stderr | 44 seconds.  [orig: 2020-09-25 06:36:45.495150 recv: 2020-09-25 06:36:01.972873]
weblate_1   | celery-translate stderr | 
weblate_1   | celery-backup stderr | [2020-09-25 06:36:53,825: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-backup stderr | 52 seconds.  [orig: 2020-09-25 06:36:53.275485 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-backup stderr 
celery-notify stderr | [2020-09-18 09:33:37,127: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-notify stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,127: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,942: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[ea475680-12b2-457e-b173-69794d161dd7]  
uwsgi stderr | [pid: 703|app: 0|req: 497/3082] 178.62.34.228 () {56 vars in 1323 bytes} [Fri Sep 18 09:29:45 2020] GET /translate/onec/r2-x-b/fr_CD/?checksum=450fc80ce224094d => generated 0 bytes in 232671 msecs (HTTP/1.0 500) 6 headers in 0 bytes (0 switches on core 0)
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/fields/related_descriptors.py", line 904, in get_queryset
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/manager.py", line 147, in get_queryset
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 191, in __init__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 227, in __init__
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 192, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 172, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/middleware.py", line 73, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 192, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 172, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 143, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 97, in render_compressed
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 67, in render_offline
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/cache.py", line 61, in get_offline_hexdigest
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/cache.py", line 22, in get_hexdigest
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/encoding.py", line 79, in smart_bytes
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/encoding.py", line 99, in force_bytes
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 312, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 446, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/base.py", line 87, in reverse
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/resolvers.py", line 654, in _reverse_with_prefix
uwsgi stderr |   File "/usr/lib/python3.7/re.py", line 183, in search
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 312, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
2020-09-18 09:34:04,528 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684648632 for <Subprocess at 140476684916888 with name uwsgi in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 446, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/base.py", line 87, in reverse
celery-main stderr | [2020-09-18 09:34:59,639: INFO/MainProcess] Received task: weblate.wladmin.tasks.configuration_health_check[ed81a207-35a7-4b5c-9c18-30f87f960908]  
celery-main stderr | [2020-09-18 09:34:59,668: INFO/MainProcess] Received task: weblate.accounts.tasks.cleanup_social_auth[72f44bba-17b3-40ab-8314-7008afeaf68e]  
celery-main stderr | [2020-09-18 09:34:59,673: INFO/MainProcess] Received task: weblate.trans.tasks.commit_pending[a699fe20-02c4-4930-88c4-8f9161f99901]  
celery-main stderr | [2020-09-18 09:34:59,727: INFO/MainProcess] Received task: weblate.accounts.tasks.cleanup_auditlog[0591408c-effb-450b-b3c1-345e30ba47e1]  
celery-main stderr | [2020-09-18 09:35:00,894: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[78aae120-1d86-4982-864a-6a08b72855ff]  
celery-backup stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,990: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,990: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | ERROR Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | [2020-09-18 09:35:23,363: ERROR/ForkPoolWorker-3] Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.accounts.tasks.cleanup_social_auth of weblate at 0x7ffae35337b8>, 'task_id': '72f44bba-17b3-40ab-8314-7008afeaf68e', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad9596308>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:24,065: ERROR/ForkPoolWorker-3] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.accounts.tasks.cleanup_social_auth of weblate at 0x7ffae35337b8>, 'task_id': '72f44bba-17b3-40ab-8314-7008afeaf68e', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad9596308>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:23,630: ERROR/ForkPoolWorker-2] Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.trans.tasks.commit_pending of weblate at 0x7ffae35337b8>, 'task_id': 'a699fe20-02c4-4930-88c4-8f9161f99901', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad8ef1a48>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:24,994: ERROR/ForkPoolWorker-2] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.trans.tasks.commit_pending of weblate at 0x7ffae35337b8>, 'task_id': 'a699fe20-02c4-4930-88c4-8f9161f99901', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad8ef1a48>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | ERROR Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | psycopg2.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | 
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr | 
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
celery-main stderr |     R = retval = fun(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
celery-main stderr |     return self.run(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/tasks.py", line 40, in cleanup_social_auth
celery-main stderr |     for partial in Partial.objects.iterator():
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 360, in _iterator
celery-main stderr |     yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 53, in __iter__
celery-main stderr |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
celery-main stderr |     cursor = self.connection.chunked_cursor()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 271, in chunked_cursor
celery-main stderr |     self._named_cursor_idx,
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
celery-main stderr |     self.ensure_connection()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | django.db.utils.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
2020-09-18 09:35:25,600 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684580848 for <Subprocess at 140476684916216 with name celery-main in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])
celery-main stderr | [2020-09-18 09:35:24,164: ERROR/ForkPoolWorker-3] Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | psycopg2.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | 
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr | 
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
celery-main stderr |     R = retval = fun(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
celery-main stderr |     return self.run(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/tasks.py", line 40, in cleanup_social_auth
celery-main stderr |     for partial in Partial.objects.iterator():
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 360, in _iterator
celery-main stderr |     yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 53, in __iter__
celery-main stderr |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
celery-main stderr |     cursor = self.connection.chunked_cursor()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 271, in chunked_cursor
celery-main stderr |     self._named_cursor_idx,
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
celery-main stderr |     self.ensure_connection()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | django.db.utils.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
2020-09-18 09:35:30,965 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684581576 for <Subprocess at 140476684916440 with name celery-notify in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])

My settings-override.py,

REQUIRE_LOGIN = False
INSTALLED_APPS = [
    # "o_customization",
    # Weblate apps on top to override Django locales and templates
    "weblate.addons",
    "weblate.auth",
    "weblate.checks",
    "weblate.formats",
    "weblate.glossary",
    "weblate.machinery",
    "weblate.trans",
    "weblate.lang",
    "weblate.langdata",
    "weblate.memory",
    "weblate.screenshots",
    "weblate.fonts",
    "weblate.accounts",
    "weblate.utils",
    "weblate.vcs",
    "weblate.wladmin",
    "weblate",
    # Optional: Git exporter
    "weblate.gitexport",
    # Standard Django modules
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "django.contrib.admin.apps.SimpleAdminConfig",
    "django.contrib.admindocs",
    "django.contrib.sitemaps",
    "django.contrib.humanize",
    # Third party Django modules
    "social_django",
    "crispy_forms",
    "compressor",
    "rest_framework",
    "rest_framework.authtoken",
    "django_filters",
]
REST_FRAMEWORK = {
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
    "DEFAULT_PERMISSION_CLASSES": [
        # Require authentication for login required sites
        "rest_framework.permissions.IsAuthenticated"
        if REQUIRE_LOGIN
        else "rest_framework.permissions.IsAuthenticatedOrReadOnly"
    ],
    "DEFAULT_AUTHENTICATION_CLASSES": (
        "rest_framework.authentication.TokenAuthentication",
        "weblate.api.authentication.BearerAuthentication",
        "rest_framework.authentication.SessionAuthentication",
    ),
    "DEFAULT_THROTTLE_CLASSES": (
        "weblate.api.throttling.UserRateThrottle",
        "weblate.api.throttling.AnonRateThrottle",
    ),
    "DEFAULT_THROTTLE_RATES": {"anon": "100/day", "user": "25000/hour"},
    "DEFAULT_PAGINATION_CLASS": ("rest_framework.pagination.PageNumberPagination"),
    "PAGE_SIZE": 500,
    "VIEW_DESCRIPTION_FUNCTION": "weblate.api.views.get_view_description",
    "UNAUTHENTICATED_USER": "weblate.auth.models.get_anonymous",
}
CHECK_LIST += (
  # "o_customization.checks.PunctuationSpacingCheckO",
)
AUTOFIX_LIST += (
  # "weblate.trans.autofixes.chars.ReplaceTrailingDotsWithEllipsis",
  # "o_customization.autofix.PunctuationSpacingAutoFixO"
)

@nijel
Copy link
Member

nijel commented Sep 29, 2020

It still seems to be out of memory issues. Can you look at individual processes how much of them is there and which consume the memory?

@stale
Copy link

stale bot commented Oct 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix Nobody will work on this. label Oct 11, 2020
@nijel nijel added this to the 4.3.1 milestone Oct 20, 2020
@nijel nijel added bug Something is broken. and removed question This is more a question for the support than an issue. wontfix Nobody will work on this. labels Oct 20, 2020
@nijel
Copy link
Member

nijel commented Oct 20, 2020

I've identified memory issue in https://github.com/WeblateOrg/translation-finder/, it is fixed in the 2.4 release and Weblate 4.3.1 will require that.

@nijel nijel closed this as completed Oct 20, 2020
@nijel nijel self-assigned this Oct 20, 2020
@github-actions
Copy link

Thank you for your report, the issue you have reported has just been fixed.

  • In case you see a problem with the fix, please comment on this issue.
  • In case you see a similar problem, please open a separate issue.
  • If you are happy with the outcome, consider supporting Weblate by donating.

@mareksuscak
Copy link
Contributor

mareksuscak commented Oct 21, 2020

EDIT: Updated to bleeding edge 4.3.1 and so far so good! Workers are not going down anymore.

Would this issue explain Weblate 4.3 going down after a while and all workers constantly restarting? That doesn't seem normal and this last release has been somewhat buggy.

image

image

@nijel
Copy link
Member

nijel commented Oct 21, 2020

Quite likely - there is a memory limit set for the workers and with the leak it is more likely it will reach it.

@tipa
Copy link
Contributor

tipa commented Oct 23, 2023

@nijel I am running on a shared hosting environment with limited RAM per user (1.5 GB). I understand that Weblate states 3GB as minimal requirement but I hope to be able to run Weblate nonetheless. The requirements seem to be quite high for just a small website.
I have set WEBLATE_WORKERS = 1 in settings.py but celery is still launching several tasks for each worker, why?
I had to set CELERY_WORKER_CONCURRENCY = 1 in order to limit the tasks per worker to two (one main celery task and one worker task, as far as I understand). Only then the celery tasks are not killed by the system for exceeding the RAM limit.
Additional settings I have set:

CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 100000 # normally set to 200000 # in kb
CELERY_WORKER_MAX_TASKS_PER_CHILD = 1 # default: no limit

But those tasks still seem to consume up to 142MB per task, as long as I am interpreting these numbers correctly. Is it normal that these tasks require so much RAM even though Weblate is completely idle / freshly set up?
image

Even when Weblate is running with CELERY_WORKER_CONCURRENCY = 1, as soon as I try to create a component based on a Git Repository, the app gets killed:

[2023-10-23 17:59:32 +0200] [17606] [INFO] Using worker: sync
[2023-10-23 17:59:32 +0200] [17609] [INFO] Booting worker with pid: 17609
[2023-10-23 18:19:34 +0200] [17606] [CRITICAL] WORKER TIMEOUT (pid:17609)
[2023-10-23 18:19:35 +0200] [17606] [ERROR] Worker (pid:17609) was sent SIGKILL! Perhaps out of memory?

This is my celery configuration for supervisord:

[group:celery]
programs=celery-backup,celery-celery,celery-memory,celery-notify,celery-translate,celery-beat
 
[program:celery-backup]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'backup@%%h' --queues=backup
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-celery]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'celery@%%h' --queues=celery
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-memory]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'memory@%%h' --queues=memory
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-notify]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'notify@%%h' --queues=notify
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-translate]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'translate@%%h' --queues=translate
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-beat]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery beat
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

EDIT: I just see that the WEBLATE_WORKERS option might only be for docker images, which is why it wasn't working

@nijel
Copy link
Member

nijel commented Oct 24, 2023

@bilalcorbacioglu Is it really necessary to post each of your posts to multiple places? Posting unrelated questions to issues that have been resolved years ago is not really helping anybody. See #10251

@WeblateOrg WeblateOrg locked as resolved and limited conversation to collaborators Oct 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something is broken.
Projects
None yet
Development

No branches or pull requests

4 participants