Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nuclei memory leak in stream mode #3043

Closed
yaabdala opened this issue Dec 14, 2022 · 12 comments · Fixed by projectdiscovery/filekv#2 or #3052
Closed

Nuclei memory leak in stream mode #3043

yaabdala opened this issue Dec 14, 2022 · 12 comments · Fixed by projectdiscovery/filekv#2 or #3052
Assignees
@Mzack9999
Labels
Priority: High After critical issues are fixed, these should be dealt with before any further issues. Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Milestone
nuclei v2.8.4

Comments

@yaabdala
Copy link

Nuclei version: 2.8.3

Current Behavior:

Nuclei hangs or quits unexpectedly. Using the -hm flag I have a generated dump file which I have attached
nuclei-stacktrace.dump.txt

This seems to happen with specific hosts. I have a handful of dns names that I ran through dnsx and httpx before feeding into nuclei.

Expected Behavior:

I expect Nuclei to log errors when scanning a group of hosts. If one host fails or hangs, it should continue scanning the rest. Failing that, it should print which host failed with which template

Steps To Reproduce:

go/bin/nuclei -l targets_ab.txt -v -hm -stats -ni -stream

Unfortunately I cannot share the hosts scanned but I can answer that I am scanning 5 hosts, some of which are WAF protected in AWS. Only 1 returns 200 from httpx, the others have redirects for authentication

Anything else:
@yaabdala yaabdala added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Dec 14, 2022
@Mzack9999
Copy link
Member

@yaabdala could you confirm if you are able to complete the scan without the -stream flag?

@Mzack9999 Mzack9999 added the Type: Question A query or seeking clarification on parts of the spec. Probably doesn't need the attention of all. label Dec 14, 2022
@yaabdala
Copy link
Author

Scan does not complete when the -stream flag is left off. It just fails slower

@tarunKoyalwar
Copy link
Member

Data from nuclei-stacktrace.txt

goroutine 33 [running]:
github.com/projectdiscovery/nuclei/v2/pkg/utils/monitor.glob..func1(0x0?)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/utils/monitor/monitor.go:93 +0x50
github.com/projectdiscovery/nuclei/v2/pkg/utils/monitor.(*Agent).monitorWorker(0xc0007f15c0)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/utils/monitor/monitor.go:71 +0xc2
github.com/projectdiscovery/nuclei/v2/pkg/utils/monitor.NewStackMonitor.func1()
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/utils/monitor/monitor.go:43 +0xb1
created by github.com/projectdiscovery/nuclei/v2/pkg/utils/monitor.NewStackMonitor
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/utils/monitor/monitor.go:37 +0xf8

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x44ecb1?)
	/usr/lib/golang/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x145ddf9?)
	/usr/lib/golang/src/sync/waitgroup.go:136 +0x52
github.com/remeh/sizedwaitgroup.(*SizedWaitGroup).Wait(...)
	/home/ouser/go/pkg/mod/github.com/remeh/sizedwaitgroup@v1.0.0/sizedwaitgroup.go:83
github.com/projectdiscovery/nuclei/v2/pkg/core.(*WorkPool).Wait(0xc0014f1ad0)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/core/workpool.go:46 +0x2b
github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).ExecuteWithOpts(0xc001002ea0, {0xc26afd4000?, 0x20683b3?, 0xc228f7eaa8?}, {0x2d45738?, 0xc0014f07e0}, 0x0?)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/core/execute.go:58 +0x33c
github.com/projectdiscovery/nuclei/v2/internal/runner.(*Runner).executeTemplatesInput(0xc000740d80, 0xc0001545a0, 0xc001002ea0)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/internal/runner/runner.go:566 +0x515
github.com/projectdiscovery/nuclei/v2/internal/runner.(*Runner).runStandardEnumeration(_, {{0x0, 0x0}, {0x0, 0x0}, {{0x0, 0x0}, {{0x0, 0x0}}, {{0x0, ...}}, ...}, ...}, ...)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/internal/runner/enumerate.go:33 +0x6d
github.com/projectdiscovery/nuclei/v2/internal/runner.(*Runner).RunEnumeration(0xc000740d80)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/internal/runner/runner.go:457 +0x16c8
main.main()
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/cmd/nuclei/main.go:101 +0x797

goroutine 12 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0000e7420)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc7
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c

goroutine 24 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc000003d40)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x4ea

goroutine 25 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc000003d40)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x52a

goroutine 26 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc000003d40)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x598

goroutine 27 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc000003d40)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5d6

goroutine 28 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000a040e0)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc7
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c

goroutine 29 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0000d1520)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x4ea

goroutine 30 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0000d1520)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x52a

goroutine 31 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0000d1520)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x598

goroutine 32 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0000d1520)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5d6

goroutine 43 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000463500)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc7
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c

goroutine 44 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc001002340)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x4ea

goroutine 45 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc001002340)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x52a

goroutine 46 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc001002340)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x598

goroutine 47 [select, 2 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc001002340)
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/ouser/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5d6

goroutine 13 [select]:
github.com/karlseguin/ccache.(*Cache).worker(0xc0005960f0)
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:167 +0xd1
created by github.com/karlseguin/ccache.(*Cache).restart
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:136 +0xf8

goroutine 14 [select, 2 minutes]:
github.com/karlseguin/ccache.(*Cache).worker(0xc000596140)
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:167 +0xd1
created by github.com/karlseguin/ccache.(*Cache).restart
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:136 +0xf8

goroutine 15 [select, 2 minutes]:
github.com/karlseguin/ccache.(*Cache).worker(0xc000596190)
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:167 +0xd1
created by github.com/karlseguin/ccache.(*Cache).restart
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:136 +0xf8

goroutine 16 [select, 2 minutes]:
github.com/karlseguin/ccache.(*Cache).worker(0xc0005961e0)
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:167 +0xd1
created by github.com/karlseguin/ccache.(*Cache).restart
	/home/ouser/go/pkg/mod/github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:136 +0xf8

goroutine 66 [select]:
github.com/projectdiscovery/ratelimit.(*Limiter).run(0xc0014f16b0)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.2/ratelimit.go:25 +0xc5
created by github.com/projectdiscovery/ratelimit.New
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.2/ratelimit.go:51 +0x10d

goroutine 68 [syscall, 2 minutes]:
os/signal.signal_recv()
	/usr/lib/golang/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	/usr/lib/golang/src/os/signal/signal.go:151 +0x2a

goroutine 69 [chan receive, 2 minutes]:
main.main.func2()
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/cmd/nuclei/main.go:87 +0x6e
created by main.main
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/cmd/nuclei/main.go:86 +0x78a

goroutine 82 [syscall, 2 minutes]:
syscall.Syscall(0x0, 0x0, 0xc00029a000, 0x1000)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:20 +0x5
syscall.read(0xc0000ae060?, {0xc00029a000?, 0x424e25?, 0x3e8d880?})
	/usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:696 +0x4d
syscall.Read(...)
	/usr/lib/golang/src/syscall/syscall_unix.go:188
internal/poll.ignoringEINTRIO(...)
	/usr/lib/golang/src/internal/poll/fd_unix.go:794
internal/poll.(*FD).Read(0xc0000ae060?, {0xc00029a000?, 0x1000?, 0x1000?})
	/usr/lib/golang/src/internal/poll/fd_unix.go:163 +0x285
os.(*File).read(...)
	/usr/lib/golang/src/os/file_posix.go:31
os.(*File).Read(0xc000010010, {0xc00029a000?, 0xc00029c000?, 0x0?})
	/usr/lib/golang/src/os/file.go:119 +0x5e
bufio.(*Reader).fill(0xc000539778)
	/usr/lib/golang/src/bufio/bufio.go:106 +0x103
bufio.(*Reader).ReadRune(0xc000539778)
	/usr/lib/golang/src/bufio/bufio.go:299 +0x25
github.com/projectdiscovery/clistats.(*Statistics).internalRead.func1()
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.9/clistats.go:157 +0x165
created by github.com/projectdiscovery/clistats.(*Statistics).internalRead
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.9/clistats.go:149 +0x56

goroutine 83 [select]:
github.com/projectdiscovery/clistats.(*Statistics).eventLoop(0xc0005960a0, 0x0?)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.9/clistats.go:132 +0x195
created by github.com/projectdiscovery/clistats.(*Statistics).Start
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.9/clistats.go:118 +0xbf

goroutine 77735 [select]:
net.(*netFD).connect.func2()
	/usr/lib/golang/src/net/fd_unix.go:118 +0x86
created by net.(*netFD).connect
	/usr/lib/golang/src/net/fd_unix.go:117 +0x37b

goroutine 24096 [IO wait]:
internal/poll.runtime_pollWait(0x7fc643dccac0, 0x77)
	/usr/lib/golang/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc21b7b1580?, 0x0?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitWrite(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).WaitWrite(...)
	/usr/lib/golang/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc21b7b1580, {0x2d47508?, 0xc21b6e7ec0}, {0xc000b86458?, 0x41b645?}, {0x2d39380?, 0xc21ab52440?})
	/usr/lib/golang/src/net/fd_unix.go:141 +0x716
net.(*netFD).dial(0xc21b7b1580, {0x2d47508, 0xc21b6e7ec0}, {0x2d4b438?, 0x0?}, {0x2d4b438?, 0xc21a91fe60}, 0x49507e?)
	/usr/lib/golang/src/net/sock_posix.go:149 +0x394
net.socket({0x2d47508, 0xc21b6e7ec0}, {0x205d1ba, 0x3}, 0x2, 0x1, 0x0?, 0x48?, {0x2d4b438, 0x0}, ...)
	/usr/lib/golang/src/net/sock_posix.go:70 +0x2b2
net.internetSocket({0x2d47508, 0xc21b6e7ec0}, {0x205d1ba, 0x3}, {0x2d4b438, 0x0}, {0x2d4b438, 0xc21a91fe60}, 0xc21ad2aa20?, 0x0, ...)
	/usr/lib/golang/src/net/ipsock_posix.go:142 +0xf8
net.(*sysDialer).doDialTCP(0xc21b7b1500, {0x2d47508, 0xc21b6e7ec0}, 0x0, 0x1d560c0?)
	/usr/lib/golang/src/net/tcpsock_posix.go:65 +0xa5
net.(*sysDialer).dialTCP(0x3e9037d?, {0x2d47508?, 0xc21b6e7ec0?}, 0xc000b866e8?, 0x47affd?)
	/usr/lib/golang/src/net/tcpsock_posix.go:61 +0x59
net.(*sysDialer).dialSingle(0xc21b7b1500, {0x2d47508, 0xc21b6e7ec0}, {0x2d42800?, 0xc21a91fe60})
	/usr/lib/golang/src/net/dial.go:583 +0x28b
net.(*sysDialer).dialSerial(0xc21b7b1500, {0x2d47508, 0xc21b6e7ec0}, {0xc21ac968d0?, 0x1, 0x20601c3?})
	/usr/lib/golang/src/net/dial.go:551 +0x312
net.(*Dialer).DialContext(0xc000514a80, {0x2d47508, 0xc21b6e7ec0}, {0x205d1ba, 0x3}, {0xc21ad2aa10, 0x10})
	/usr/lib/golang/src/net/dial.go:428 +0x736
crypto/tls.dial({0x2d474d0?, 0xc0000460a8?}, 0xc000514a80, {0x205d1ba, 0x3}, {0xc21ad2aa10, 0x10}, 0xc21b686c00)
	/usr/lib/golang/src/crypto/tls/tls.go:133 +0x1a9
crypto/tls.DialWithDialer(...)
	/usr/lib/golang/src/crypto/tls/tls.go:117
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).dial(0xc0006a05c0, {0x2d474d0, 0xc0000460a8}, {0x205d1ba, 0x3}, {0xc22ff94906, 0x15}, 0x1, 0x0, 0xc44b482180, ...)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.0.19/fastdialer/dialer.go:222 +0x13b8
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).DialTLSWithConfig(...)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.0.19/fastdialer/dialer.go:127
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).DialTLS(0xc0006a05c0, {0x2d474d0, 0xc0000460a8}, {0x205d1ba, 0x3}, {0xc22ff94906, 0x15})
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.0.19/fastdialer/dialer.go:116 +0x10a
github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeRequestWithPayloads(0xc22ff20c00, 0xc000b87608, {0xc22ff94906, 0x15}, {0x0, 0x0}, {0x0, 0x0}, 0x1, 0xc21beed5c0, ...)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/protocols/network/request.go:114 +0x108
github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeAddress(0xc22ff20c00, 0xc44b5caa68, {0xc22ff94906, 0x15}, {0x0, 0x0}, {0x0, 0x0}, 0xf?, 0xc21beed4d0, ...)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/protocols/network/request.go:95 +0x505
github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).ExecuteWithResults(0xc22ff20c00, 0xc0009d39c0, 0xc21beed4a0?, 0xc21beed4d0?, 0xc21beed500?)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/protocols/network/request.go:60 +0x3c9
github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute(0xc22ff96420, 0xc0009d3960)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/protocols/common/executer/executer.go:80 +0x3b0
github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeSelfContainedTemplateWithInput(0xc001002ea0, 0xc22ff474a0, 0xc24af26f68)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/core/execute.go:64 +0xa6
github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).ExecuteWithOpts.func1(0xc21b1aed80?)
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/core/execute.go:51 +0x7e
created by github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).ExecuteWithOpts
	/home/ouser/go/pkg/mod/github.com/projectdiscovery/nuclei/v2@v2.8.3/pkg/core/execute.go:45 +0xd9

@vzamanillo
Copy link
Contributor

vzamanillo commented Dec 15, 2022 •
edited
Loading

I was able to reproduce this even using the dev branch, I think there is a memory leak somewhere (15GB avg before quit)

imagen

@tarunKoyalwar
Copy link
Member

@vzamanillo @yaabdala can you run nuclei with below format and print the output . This will give us more details regarding the crash.

/usr/bin/time -l -h nuclei -l targets_ab.txt -v -hm -stats -ni -stream

@vzamanillo
Copy link
Contributor

vzamanillo commented Dec 15, 2022 •
edited
Loading

I think the leak is located here

https://github.com/projectdiscovery/nuclei/blob/main/v2/pkg/core/inputs/hybrid/hmap.go#L250

It's a bug on filekv at

https://github.com/projectdiscovery/filekv/blob/main/file.go#L166

due to a buffer memory allocation.

@yaabdala
Copy link
Author

I can't pass time the -l -h flags but I ran

/usr/bin/time go/bin/nuclei -l targets_ab.txt -v -hm -stats -ni -stream

result

538.29user 40.67system 2:50.15elapsed 340%CPU (0avgtext+0avgdata 46394588maxresident)k 0inputs+200outputs (0major+574574minor)pagefaults 0swaps

I also concur with @vzamanillo that there seems to be a memory leak when using the -stream flag.

@tarunKoyalwar
Copy link
Member

@yaabdala If it is memory leak pls share pprof file using option -profile-mem pprof.out

@vzamanillo
Copy link
Contributor

@tarunKoyalwar it's a confirmed memory leak check my previous comment.

@yaabdala
Copy link
Author

pprof.out.txt

Here is a copy of the memory dump for the run. Had to change the extension to .txt due to github upload restrictions

@vzamanillo vzamanillo mentioned this issue Dec 15, 2022
Merged
@yaabdala
Copy link
Author

yaabdala commented Dec 15, 2022 •
edited
Loading

without the stream flag, I don't see a memory leak. In fact, memory utilization is very low. Any ideas why we are seeing the failures?

[0:00:55] | Templates: 4473 | Hosts: 5 | RPS: 85 | Matched: 5 | Errors: 656 | Requests: 4696/30340 (15%)
[WRN] Could not make network request for <HOST>:465: could not connect to server request: no address found for host
[WRN] Could not make network request for <HOST>:21: could not connect to server request: could not connect to any address found for host

[WRN] Could not make network request for <HOST>:25: could not connect to server request: could not connect to any address found for host

[WRN] Could not make network request for <HOST>:23: could not connect to server request: could not connect to any address found for host

[WRN] Could not make network request for <HOST>:8080: could not connect to server request: could not connect to any address found for host

[WRN] Could not make network request for <HOST>:8009: could not connect to server request: could not connect to any address found for host

[WRN] Could not make network request for <HOST>:23: could not connect to server request: could not connect to any address found for host
[WRN] Could not make network request for <HOST>:3310: could not connect to server request: could not connect to any address found for host
[WRN] Could not make network request for <HOST>:139: could not connect to server request: could not connect to any address found for host
[0:01:00] | Templates: 4473 | Hosts: 5 | RPS: 78 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[0:01:05] | Templates: 4473 | Hosts: 5 | RPS: 72 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[0:01:10] | Templates: 4473 | Hosts: 5 | RPS: 67 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[0:01:15] | Templates: 4473 | Hosts: 5 | RPS: 62 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[0:01:20] | Templates: 4473 | Hosts: 5 | RPS: 58 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[0:01:25] | Templates: 4473 | Hosts: 5 | RPS: 55 | Matched: 5 | Errors: 665 | Requests: 4705/30340 (15%)
[WRN] Could not make network request for <HOST>:465: could not connect to server request: no address found for host
[0:01:28] | Templates: 4473 | Hosts: 5 | RPS: 53 | Matched: 5 | Errors: 666 | Requests: 4706/30340 (15%)
profile: memory profiling disabled, pprof.out
477.90user 40.82system 2:11.37elapsed 394%CPU (0avgtext+0avgdata 38185276maxresident)k
0inputs+496outputs (0major+724639minor)pagefaults 0swaps

@vzamanillo
Copy link
Contributor

vzamanillo commented Dec 16, 2022 •
edited
Loading

Also, using the filekv default options allocates an excessive memory amount (2147483647) due to the bloom filter initialization for dedupe.

https://github.com/projectdiscovery/nuclei/blob/main/v2/pkg/core/inputs/hybrid/hmap.go#L56

@vzamanillo vzamanillo mentioned this issue Dec 16, 2022
Closed
4 tasks
@ehsandeep ehsandeep added this to the nuclei v2.8.4 milestone Dec 16, 2022
@Mzack9999 Mzack9999 reopened this Dec 16, 2022
@Mzack9999 Mzack9999 self-assigned this Dec 16, 2022
@Mzack9999 Mzack9999 changed the title Nuclei hangs and sometimes quits unexpectedly Nuclei memory leak in stream mode Dec 16, 2022
@Mzack9999 Mzack9999 added Priority: High After critical issues are fixed, these should be dealt with before any further issues. and removed Type: Question A query or seeking clarification on parts of the spec. Probably doesn't need the attention of all. labels Dec 16, 2022
@Mzack9999 Mzack9999 linked a pull request Dec 16, 2022 that will close this issue
Fix memory leak on hmap with LRU #3052
Merged
4 tasks
@vzamanillo vzamanillo mentioned this issue Dec 16, 2022
Closed
@ehsandeep ehsandeep added the Status: Completed Nothing further to be done with this issue. Awaiting to be closed. label Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Mzack9999 Mzack9999

Labels
Priority: High After critical issues are fixed, these should be dealt with before any further issues. Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Milestone
nuclei v2.8.4
5 participants
@ehsandeep @vzamanillo @Mzack9999 @tarunKoyalwar @yaabdala