-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM killed when using the nuclei SDK with the standard templates #4756
Comments
Have you tried reproducing with the latest version: https://github.com/projectdiscovery/nuclei/releases/tag/v3.1.10 ? |
@stan-threatmate , there was a minor change related to js pooling and upgrade in other pd dependencies so please try with latest version or even
|
I've been running it all day today with the latest version v3.1.10 but I see the same issues. Also added Also your suggestions in the comment above contradict this document which I used to set the above options:
Can you please provide a recommendation on what settings effect the memory consumption the most and what settings effect the speed of execution? For example I've noticed the rate limit option doesn't really play much of a role in the SDK as reported by the stats which print the RPS. I assume the RPS is the request per second as defined by the rate limit? I'll do some runs with your suggestion: 25 template and host concurrency. I wish there was a way to understand the system resource utilization based on the settings so we can plan for it based on the number of hosts. |
Here is a pprof from a successful run on a smaller scale:
Note on the second line how much the
Also this stat is printed after nuclei has finished but it shows 93% and the stat never stops printing. |
A profile of a more intense run:
|
And this uses what you suggested 25 template/host concurrency:
|
Actually your proposal of having 25 concurrent hosts/templates worked on one of my test setups. I setup a constrained memory container with 2048MB RAM and aggressive GC settings: Then around the 75% complete it shot up again this time staying for a very long time at 2GB and the CPU really hurting at 1500% trying to free all this memory. It was successful and completed at the end. My theory is that some templates are allocating a ton of memory and if the concurrency settings are above a certain threshold it can lead to the allocation rate surpassing the ability of the GC to free the memory which ultimately leads to an OOM kill. The only saving grace would be good amount of free memory and/or fast CPUs that can help the GC free up memory faster. But we really need guidance on the performance characteristics of nuclei: what is the RAM and CPU requirements for X many hosts and Y many templates etc. Is there a way to know which templates use the most memory? Can we measure the CPU/memory of individual templates? That would be a very good metric to know. If I want to speed things up I'd like to be able to run efficient templates faster but slow down on the memory heavy ones in order to not run out of memory. Some of the big allocations are around the raw requests and responses. |
Another update - using 25/25 for host/template concurrency when scanning 360 targets still resulted in OOM kill but it ran for a significantly longer time. I will set the garbage collection to aggressive values and try again: |
@stan-threatmate , although it is helpful in production but tuning GC while debugging memory leaks might not help so i would recommending just to try out with normal options . because as you already know go does not immediately release memory but it gradually releases it in hope of reusing it instead of allocating it again and again . that is why tuning GC aggresively would only cause more cpu utilization without any direct output (especially in this case ) Just now i have added more docs on how nuclei consumes resources and all the factors involved based on your suggestion at https://docs.projectdiscovery.io/tools/nuclei/mass-scanning-cli From the profile details you have shared it looks like these are not the actual inflection points
Also looking at above profile data i can only tell that the top functions using heap as shown in above profiles are expected . generator.MergeMaps , generateRawRequest etc contain If you think its related to particular set of templates i would suggest splitting templates and running different scan
^ This is one of effective strategy i used which worked when i fixed memory leaks recently in v3.1.8-10 Other/Alternative strategy is to continiously capture snapshots and nuclei process memory (using memstats or manually via bash script using PID) . subtracting profiles between normal and sudden spike using I will try to reproduce this using CLI with 800 targets. Finally if you want to customize any execution behaviour or use your own logic . i would suggest taking a look at |
btw nuclei-docs repo is deprecated and latest docs are available at https://docs.projectdiscovery.io/introduction |
@tarunKoyalwar thank you for looking into this issue! I have a question about Also about the About the |
@stan-threatmate FYI , we were able to reproduce this some time ago and working on locating and fixing the vulnerable code |
@tarunKoyalwar thank you! |
The issue will be fixed as part of #4800 |
@stan-threatmate , can you try running scan using sdk by disabling these 4 templates ? you can use http/cves/2019/CVE-2019-17382.yaml
http/cves/2023/CVE-2023-24489.yaml
http/fuzzing/header-command-injection.yaml
http/fuzzing/wordpress-weak-credentials.yaml |
@tarunKoyalwar I removed the templates you mention and my first test scan finished successfully. Memory looks great. Next I am running a large scan over 400 hosts but it will take 15h to complete so I'll report tomorrow. I also used more aggressive settings:
|
@stan-threatmate The issue is about high parallelism in bruteforce templates, which causes a lot of buffer allocations to read http responses (up to default 10mb). To mitigate the issue a generic memory monitor mechanism, has been implemented in #4833 (when the global RAM occupation is above |
@Mzack9999 thank you! How is the RAM limit determined? Is it based on the free memory or the total memory? Can we configure the limits (75% and 5 threads) in the SDK? Update: Second update: |
Nuclei version:
3.1.8
Current Behavior:
I run the nuclei SDK as part of a binary that is deployed in a linux container (alpine) with memory limits of 8GB and 16GB. I use the standard templates. In both cases it gets OOM killed. Here are the settings I specify.
I tried this with 115 and 380 hosts and both are having memory issues. What is causing the high memory utilization? I am saving the results from the nuclei scan in a list. Could the results be so large that they fill in the memory?
I run nuclei like this:
Expected Behavior:
The nuclei SDK should trivially handle scanning hosts with the above settings. It will be great to have an example of the SDK settings that match the default nuclei cli scan settings.
What would be the equivalent settings for the SDK?
Additionally what settings in the SDK control the memory utilization? It will be good to document those as well.
Steps To Reproduce:
Use the above settings and set up a scan. Watch it take a lot of memory over time. Better if you use 115 (or more) web sites.
Anything else:
The text was updated successfully, but these errors were encountered: