Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

uwsgi service eating up all memory #5174

Closed
1 task done
erciccione opened this issue Jan 9, 2021 · 5 comments
Closed
1 task done

uwsgi service eating up all memory #5174

erciccione opened this issue Jan 9, 2021 · 5 comments

Comments

@erciccione
Copy link

erciccione commented Jan 9, 2021 •
edited

Describe the issue

Suddenly, the uwsgi service started to consume a lot of memory. To the point where all RAM + swap space is full. Weblate becomes unavailable and the only way to resolve is to stop uwsgi and nginx.

I already tried

Describe the steps you tried to solve the problem yourself.

  • I've read and searched the docs and did not find the answer there.
    If you didn’t try already, try to search there what you wrote above.

Exception traceback
logs from /var/log/uwsgi/app/weblate.log

ERROR Internal Server Error: /translate/community/monero-means-money/en/
    Traceback (most recent call last):
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
        response = get_response(request)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/core/handlers/base.py", line 169, in _get_response
        response = middleware_method(request, callback, callback_args, callback_kwargs)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/middleware/csrf.py", line 313, in process_view
        return self._reject(request, REASON_BAD_TOKEN)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/middleware/csrf.py", line 149, in _reject
        response = _get_failure_view()(request, reason=reason)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/views/csrf.py", line 154, in csrf_failure
        return HttpResponseForbidden(t.render(c), content_type='text/html')
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/backends/django.py", line 61, in render
        return self.template.render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 170, in render
        return self._render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
        return self.nodelist.render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 938, in render
        bit = node.render_annotated(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
        return self.render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/loader_tags.py", line 150, in render
        return compiled_parent._render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
        return self.nodelist.render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 938, in render
        bit = node.render_annotated(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
        return self.render(context)
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/library.py", line 192, in render
        output = self.func(*resolved_args, **resolved_kwargs)
        File "/var/www/weblate-env/lib/python3.6/site-packages/weblate/trans/templatetags/translations.py", line 276, in documentation
        return get_doc_url(page, anchor, user=context["user"])
        File "/var/www/weblate-env/lib/python3.6/site-packages/django/template/context.py", line 83, in __getitem__
        raise KeyError(key)
    KeyError: 'user'

Note that ERROR Internal Server Error: /translate/community/monero-means-money/en/ happens for basically all projects and folders.

nginx logs:

A series of

2021/01/09 14:55:57 [error] 12504#12504: *15818 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.2.2, server: weblate, request: "GET /translate/community/monero-means-money/el/?checksum=1%20waitfor%20delay%20'0:0:0'%20--%20&sort_by=-priority%2Cposition HTTP/1.0", upstream: "uwsgi://127.0.0.1:8080", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"

I haven't touched my nginx configuration.

Server configuration and status

Weblate 4.4 installed with pip
@erciccione
Copy link
Author

erciccione commented Jan 9, 2021 •
edited

Taking a look at #1374 (comment) i noticed i too had this package missing. I installed it and the errors seem to be gone for now (EDIT: Errors are back. It was probably unrelated). i only get some warning when running service uwsgi status:

Jan 09 16:05:51 vagrant weblate[26811]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 56), (line:1, col:57)
Jan 09 16:05:54 vagrant weblate[26810]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 30), (line:1, col:31)
Jan 09 16:05:55 vagrant weblate[26810]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 56), (line:1, col:57)
Jan 09 16:05:57 vagrant weblate[26811]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 2), (line:1, col:3)
Jan 09 16:05:59 vagrant weblate[26810]: WARNING Handled exception: ValueError: Invalid query string.
Jan 09 16:06:02 vagrant weblate[26810]: WARNING Handled exception: ValueError: Invalid query string.
Jan 09 16:06:04 vagrant weblate[26810]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 2), (line:1, col:3)
Jan 09 16:06:11 vagrant weblate[26810]: WARNING Handled exception: ParseException: Expected end of text, found '('  (at char 0), (line:1, col:1)
Jan 09 16:06:11 vagrant weblate[26811]: WARNING Handled exception: ParseException: Expected end of text, found ')'  (at char 2), (line:1, col:3)
Jan 09 16:06:14 vagrant weblate[26810]: WARNING Handled exception: ParseException: Expected end of text, found '('  (at char 0), (line:1, col:1)

Resource usage still seems high. Keeping monitor it for now, if everything runs fine, i'll just close this issue.

@erciccione
Copy link
Author

erciccione commented Jan 9, 2021 •
edited

The RAM usage is still contantly increasing. Nginx errors (/var/log/nginx/error.log):

2021/01/09 16:55:59 [error] 26790#26790: *30426 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/font-source'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'/OTF/SourceCodePro-BoldIt.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/font-source'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'/OTF/SourceCodePro-BoldIt.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:55:59 [error] 26790#26790: *30427 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/1'"/OTF/SourceCodePro-BoldIt.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/1'"/OTF/SourceCodePro-BoldIt.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:55:59 [error] 26790#26790: *30428 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/(select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/OTF/SourceCodePro-ExtraLightIt.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/(select(0)from(select(sleep(15)))v)%2f*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*%2f/OTF/SourceCodePro-ExtraLightIt.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:55:59 [error] 26790#26790: *30431 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/@@GpgoE/OTF/SourceCodePro-BoldIt.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/@@GpgoE/OTF/SourceCodePro-BoldIt.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:00 [error] 26790#26790: *30432 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/CACHE/css/XgqRHjW9')) OR 390=(SELECT 390 FROM PG_SLEEP(15))--.3f4d6a4137f6.css" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/CACHE/css/XgqRHjW9'))%20OR%20390=(SELECT%20390%20FROM%20PG_SLEEP(15))--.3f4d6a4137f6.css HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:01 [error] 26790#26790: *30433 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/CACHE/css/0"XOR(if(now()=sysdate(),sleep(15),0))XOR"Z.895a77d42b19.css" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/CACHE/css/0"XOR(if(now()=sysdate(),sleep(15),0))XOR"Z.895a77d42b19.css HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:02 [error] 26790#26790: *30436 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/CACHE/css/ShirYt8J')) OR 67=(SELECT 67 FROM PG_SLEEP(15))--.6844a7e88625.css" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/CACHE/css/ShirYt8J'))%20OR%2067=(SELECT%2067%20FROM%20PG_SLEEP(15))--.6844a7e88625.css HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:05 [error] 26790#26790: *30443 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/VWRUhH9R') OR 117=(SELECT 117 FROM PG_SLEEP(15))--/font-source/OTF/SourceCodePro-Light.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/VWRUhH9R')%20OR%20117=(SELECT%20117%20FROM%20PG_SLEEP(15))--/font-source/OTF/SourceCodePro-Light.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:05 [error] 26790#26790: *30444 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/1XHHLGAO'; waitfor delay '0:0:15' -- /OTF/SourceCodePro-ExtraLight.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/1XHHLGAO'%3b%20waitfor%20delay%20'0:0:15'%20--%20/OTF/SourceCodePro-ExtraLight.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:06 [error] 26790#26790: *30447 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'/font-source/OTF/SourceCodePro-It.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'/font-source/OTF/SourceCodePro-It.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:06 [error] 26790#26790: *30448 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/1'"/font-source/OTF/SourceCodePro-It.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/1'"/font-source/OTF/SourceCodePro-It.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:06 [error] 26790#26790: *30449 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/vendor/1 waitfor delay '0:0:15' -- /OTF/SourceCodePro-ExtraLightIt.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/vendor/1%20waitfor%20delay%20'0:0:15'%20--%20/OTF/SourceCodePro-ExtraLightIt.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:07 [error] 26790#26790: *30450 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/@@AvZxS/font-source/OTF/SourceCodePro-It.otf" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/@@AvZxS/font-source/OTF/SourceCodePro-It.otf HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"
2021/01/09 16:56:08 [error] 26790#26790: *30455 open() "/var/www/weblate-env/lib/python3.6/site-packages/data/static/CACHE/css/output'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'.3f4d6a4137f6.css" failed (2: No such file or directory), client: 10.0.2.2, server: weblate, request: "GET /static/CACHE/css/output'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'.3f4d6a4137f6.css HTTP/1.0", host: "translate.getmonero.org", referrer: "https://translate.getmonero.org/"

nijel added a commit that referenced this issue Jan 9, 2021
@nijel
Templatetags: Fix crash when user is not present in the context
4c51a87 
This can happen on the error pages.

Issue #5174
@nijel
Copy link
Member

nijel commented Jan 9, 2021

The error in the initial comment should be addressed by 4c51a87, but I don't think it's related to the memory consumption.

Errors in #5174 (comment) should be harmless, in case you want to get more insight, configure error collection to Sentry or Rollbar.

#5174 (comment) shows HTTP errors from some bots probing invalid URLs, these don't even reach uwsgi, so it's not directly related.

But it might indicate that some bots are trying to attack your server and that might cause higher load and reveal some memory leaks (either in Weblate or in some of the libraries we use). The example uwsgi configuration is using reload-on-rss = 250. That is there merely to address an issue that Python never releases memory and some operations in Weblate are memory hungry, but it should help in this situation as well.

@erciccione
Copy link
Author

Thank you very much. The situation seem to be under control now (and reload-on-rss = 250 seems to have helped).

@github-actions
Copy link

The issue you have reported is resolved now. If you don’t feel it’s right, please follow it’s labels to get a clue and take further steps.

  • In case you see a similar problem, please open a separate issue.
  • If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nijel @erciccione