New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
uWSGI OOMKilled on Kubernetes #9562
Comments
Can you please open it as PR? It looks, like you already know the solution :) |
Sure! Just wanted to make sure you where willing to accept it 👍 |
I'm not a moderator (just a regular member of the community) but from what I see deep discussion usually opens under open PR. |
Thanks for the insights, I will open a PR 🚀 |
@hoeg To add on to what @kiblik said - For Helm in particular, we're trying to keep it at a 'generic framework for deploying DefectDojo level - no opinionated to much in any particular direction. I know we've pushed back on very specific k8s/Helm changes that pushed the Helm towards only working on a specific vendors cloud or specific tech choice (like HA vs not-HA DB). So, please keep this in mind when creating that PR. We're a project with a very broad community who deploy DefectDojo on everything from a laptop running Kali Linux to auto-scaling k8s and we try to keep a balance between those deployment choices in what we accept into the main repo. For corner cases or very vendor specific things, we'd prefer the ability to opt-in to that choice while keeping the current default. Anyway, that's how we try to balance a specific community member need vs the broader community. HTH. |
After removing the CPU limit on the iwsg container and increasing the memory limit to 3Gi, i have no more oomKill. Now my memory not go higher than 1Gi |
Seems like this OOOMKill issue is a k8s config issue. Closing this. For future readers of this thread, the best place to get advice on running DefectDojo, the OWASP Slack has a broad and active community. Info on Slack is at https://github.com/DefectDojo/django-DefectDojo?tab=readme-ov-file#community-getting-involved-and-updates |
Bug description
Deploying Defect Dojo to a Kubernetes cluster causes the uWSGI container to consume a lot of memory resulting in the node killing the pod. This is due to the unbound number of file descriptors on the node. See unbit/uwsgi#2299 for a description of the issue with uWSGI.
Steps to reproduce
Expected behavior
Expected the pod to start up and not get OOMKilled by the node.
I locally build my own container adding the
--max-fd
argument todocker/entrypoint-uwsgi.sh
and used that image in the my cluster, this resolved the issue.Deployment method (select with an
X
)Environment information
Logs
Logs from the defectdojo-django pod
note that uWSGI logs
detected max file descriptor number: 1073741816
which causes the container to use a lot of memory.Running the same deployment locally on my kind cluster i get:
where we see that
detected max file descriptor number: 1048576
. This is much lower and does not result in a OOMKilled event.Suggestion
We add the option to include the
--max-fd
argument with a configurable value to thedocker/entrypoint-uwsgi.sh
script such that it is possible to set it to set it to a lower value, e.g.1048576
.The text was updated successfully, but these errors were encountered: