Out of Memory Issue?

[haydenflinner]

Got this issue a few times today. On OSX with 16GB RAM. Calling gobuster from 10 python processes (also tried with 10 threads) with -t 5 and a 10 word list. Is it possible that there's a memory leak? My code is using only subprocess module functions that wait for the program to end before continuing.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1210e58]

goroutine 1 [running]:
main.SetupDir(0xc4200d0140, 0x1010452)
	/Users/me/.go/src/gobuster-master/main.go:563 +0x138
main.Process(0xc4200d0140)
	/Users/me/.go/src/gobuster-master/main.go:433 +0x79
main.main()
	/Users/me/.go/src/gobuster-master/main.go:824 +0x3e

Update: Memory usage shows no signs of climbing out of control before this error happens
5 minutes after last update Update: I think it's just that MakeRequest is returning nil because http.NewRequest is failing.

[haydenflinner]

If anyone runs into this, I fixed it in my case by increasing the number of files allowed to be open and thus the number of sockets that could be open. But I think that gobuster could probably handle the failure a little more gracefully, if you're itching for something to fix.

[OJ]

Thanks, I'll see if I can repro this. Interesting that you're seeing it though, as Gobuster only opens a single file (wordlist) and then concurrent TCP for the brute based on threads. I would have thought that this wouldn't be an issue without thousands of threads.

[haydenflinner]

I'm close to thousands of threads now with no issues. 250 thread pool * 5 gobuster threads --> 1250 at once.

I believe I was only running into this because OSX's default ulimit is something dumb like 512.

[OJ]

This should be good now :)