Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding memguardian + various optimizations #4833

Merged
merged 20 commits into from
Mar 14, 2024
Merged

Adding memguardian + various optimizations #4833

merged 20 commits into from
Mar 14, 2024

Conversation

Mzack9999
Copy link
Member

@Mzack9999 Mzack9999 commented Mar 5, 2024

Proposed changes

  • Introducing the memguardian component that actively monitor used memory. Activation is optional via export MEMGUARDIAN=1. If the used RAM is above the threshold (default 75% of available RAM). It reduces parallel threading of templates to either MEMGUARDIAN_THREADS if defined, or 1/5 of the total original threading. Also if MEMGUARDIAN_ALLOC is defined, it caps the maximum amount of buffers that can be allocated in the pool. When memory conditions are back to normal, the slow down mechanisms are automatically reverted.
  • collecting events within ScanContext is now optional, as it's not used anywhere and it was preventing gc to recycle large maps of data (and probably a co-cause of running nuclei with sdk may have gc problem #4836)

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Example

Closes #4756

$ MEMGUARDIAN=1 go run . -l 790targets.txt -jsc 110 -stats -tags fuzz,fuzzing,brute-force,bruteforce,brute,dos,local,code  -severity critical -rl 0
...
[INF] Templates loaded for current scan: 6
[INF] Executing 4 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 3 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 790
[0:00:06] | Templates: 6 | Hosts: 790 | RPS: 1796 | Matched: 0 | Errors: 0 | Requests: 11227/6737120 (0%)
...
[0:35:50] | Templates: 6 | Hosts: 790 | RPS: 686 | Matched: 0 | Errors: 37507 | Requests: 6417135/6737120 (95%)

@Mzack9999 Mzack9999 self-assigned this Mar 6, 2024
@Mzack9999 Mzack9999 added the Type: Enhancement Most issues will probably ask for additions or changes. label Mar 6, 2024
@Mzack9999 Mzack9999 marked this pull request as ready for review March 6, 2024 12:53
@Mzack9999 Mzack9999 merged commit b7f76cf into dev Mar 14, 2024
12 checks passed
@Mzack9999 Mzack9999 deleted the maint-memory branch March 14, 2024 22:39
@Mzack9999
Copy link
Member Author

Optional behavior - Merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement Most issues will probably ask for additions or changes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OOM killed when using the nuclei SDK with the standard templates
2 participants