Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Oncall engine using too much memory #1521

Closed
relip opened this issue Mar 9, 2023 · 2 comments · Fixed by #2884
Closed

Oncall engine using too much memory #1521

relip opened this issue Mar 9, 2023 · 2 comments · Fixed by #2884
Assignees
@vadimkerr
Labels
part:deployment/helm

Comments

@relip
Copy link

relip commented Mar 9, 2023 •
edited

I'm currently running Oncall OSS in Kubernetes cluster with Helm and it has come to my attention that Oncall engine, possibly uWSGI, is using memory more than 30GiB per pod even if I manually reduced process count from 5 to 1.

image

Also I tried to launch django app myself through python manage.py runserver it has gone down to 1GiB.

Lens exe_Y0a3ubtA07

Oncall version is 1.1.36.

I suspect there is an issue on uWSGI side?
@relip
Copy link
Author

relip commented Mar 20, 2023

After some digging, I found that uWSGI allocates memory by system's max fd size, which was 1073741816 in my case.

Since 1073741816 is the default value for containerd, this is not only helm specific issue but also affecting every docker deployment with uWSGI (tested on docker compose), the fix would be migrating over gunicorn (or anything else) or hardcoding max-fd in uwsgi.ini.

@relip relip mentioned this issue Mar 20, 2023
Open
@relip
Copy link
Author

relip commented Mar 23, 2023

Can anyone look into this issue? I'm currently manually editing oncall-engine deployment to add max-fd in uwsgi.ini every new version release. Maybe @vadimkerr?

@vadimkerr vadimkerr self-assigned this Aug 24, 2023
@vadimkerr vadimkerr mentioned this issue Aug 25, 2023
Merged
3 tasks
github-merge-queue bot pushed a commit that referenced this issue Aug 25, 2023
@vadimkerr
Prevent uWSGI from consuming too much memory (#2884)
fbfcf7b 
# What this PR does

Fixes #1521 (see
unbit/uwsgi#2299)

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vadimkerr vadimkerr

Labels
part:deployment/helm
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent uWSGI from consuming too much memory grafana/oncall
2 participants
@relip @vadimkerr