Added custom prefix, suffix functionalities and -random-agent. #56

BBerastegui · 2017-09-19T10:44:56Z

I added the option to specify custom prefix and suffix to the wordlist entries, just in case we want to try any kind of weird path combination, such as %2f or //, etc.

0xdevalias

Looks pretty good to me. May require minor changes to merge into 1.4-dev refactoring when it comes to it. Does the current CLI param flag implementation support --param format? If so, would make more sense to use --prefix and --suffix IMO.

BBerastegui · 2017-10-10T10:45:46Z

The --param format seems not to be "officially" supported, but it works if being use in the command line.

BBerastegui · 2017-10-10T18:24:17Z

I also added the -random-agent feature with the user-agent dictionary from sqlmap.

OJ · 2018-01-14T10:45:57Z

Hi @BBerastegui Thanks very much for this. I've just landed 1.4 (finally), sorry for the delay. Could you please rebase this against those changes? Thank you very much!

BBerastegui · 2018-01-16T12:04:46Z

I'm trying to do so :P

But:

libgobuster/dir.go:128: undefined: uuid.Must

I can't find what's going on here. Any clue what I'm missing?

:'(

OJ · 2018-01-16T21:28:31Z

Be sure to update your uuid lib code, that'll remove the compiler error.

BBerastegui · 2018-01-16T23:04:38Z

As a reference, for those getting:

undefined: uuid.Must error when doing go build .

You should force update of the dependencies (doing only go get in the uuid library wasn't working for me):

go get -u -v

BBerastegui · 2018-01-16T23:50:26Z

Me using Github like:

I'm sorry for this mess. Shame.

0xdevalias · 2018-01-17T01:01:11Z

libgobuster/dir.go

+	if s.RandomUserAgent && s.UserAgent == "" {
+		b, err := ioutil.ReadFile("user-agents.txt")
+		if err != nil {
+			fmt.Print(err)


You catch the error here, print it, but then execution will just continue I believe? We would probably be better returning the error here, or (maybe, but it's gross) panic'ing?

How do you suggest to return if error then?

return nil, nil ?

Normally it would be ‘return nil, er’ or similar. On my phone so can’t check the return type, may need to add a 3rd type (error) if it’s already returning 2. Then return nil, nil, error.

Can’t remember if I started implementing multierror in this or not, if so, possibly using that type might be beneficial.

Would also have to add an error check around anywhere that calls this.

Ideally we would be bubbling any errors back up to the highest point, so it can decide how to handle them (or die). Rather than the current panics we have scattered around the place.

A little divergent from initial PR, but implementing that pattern for this function would help to clear up some of the existing tech debt.

Function MakeRequest returns (*int, *int64) so it gives an error if you try to return nil, err.

I'll leave it as the rest of the returns in that function, as nil, nil

0xdevalias · 2018-01-17T01:01:38Z

libgobuster/dir.go

@@ -59,6 +61,22 @@ func MakeRequest(s *State, fullUrl, cookie string) (*int, *int64) {
 		req.Header.Set("User-Agent", s.UserAgent)
 	}

+	if s.RandomUserAgent && s.UserAgent == "" {
+		b, err := ioutil.ReadFile("user-agents.txt")


Hardcoded filename, is there a way to specify this from the CLI parameters?

0xdevalias · 2018-01-17T01:03:06Z

libgobuster/state.go

@@ -36,39 +36,42 @@ type StringSet struct {
 // Contains State that are read in from the command
 // line when the program is invoked.
 type State struct {
-	Client         *http.Client


Not sure why this is all marked as such a big block of changes. Did the whitespacing used change or something? Does this match gofmt?

This does match gofmt, but I have no idea either.

Haha, fair enough. Not a big issues

0xdevalias · 2018-02-14T10:10:24Z

I would be making the return type (*int, *int64, error) then, and returning nil, nil, err. *as alluded to above* Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: Borja Berastegui <notifications@github.com> Sent: Wednesday, February 14, 2018 9:03:24 PM To: OJ/gobuster Cc: 0xdevalias; Comment Subject: Re: [OJ/gobuster] Added custom prefix, suffix functionalities and -random-agent. (#56) @BBerastegui commented on this pull request.

________________________________ In libgobuster/dir.go<#56 (comment)>:

@@ -59,6 +61,22 @@ func MakeRequest(s *State, fullUrl, cookie string) (*int, *int64) {

req.Header.Set("User-Agent", s.UserAgent) } + if s.RandomUserAgent && s.UserAgent == "" { + b, err := ioutil.ReadFile("user-agents.txt") + if err != nil { + fmt.Print(err) Function MakeRequest returns (*int, *int64) so it gives an error if you try to return nil, err. I'll leave it as the rest of the returns in that function, as nil, nil — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#56 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAuA46VT_M-9lHlsaNnNAuUj2-wiEaVuks5tUq9sgaJpZM4PcM2Q>.

BBerastegui · 2018-02-14T10:18:29Z

GoGet must now then return one additional argument too:

func GoGet(s *State, url, uri, cookie string) (*int, *int64, error) { return MakeRequest(s, url+uri, cookie) }

Would you like to keep sending this error up to the moon?

Before:
wildcardResp, _ := GoGet(s, s.Url, fmt.Sprintf("%s", guid), s.Cookies)
After:
wildcardResp, _, err := GoGet(s, s.Url, fmt.Sprintf("%s", guid), s.Cookies)

And same in all cases where GoGet is being called?

0xdevalias · 2018-02-14T11:17:27Z

That was my intention, pushing error handling out to the ‘highest’ point where it makes logical sense to be handled. Which was also why I said it might be beyond the scope of this PR depending how much would have to change. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Borja Berastegui <notifications@github.com> Sent: Wednesday, February 14, 2018 9:18:32 PM To: OJ/gobuster Cc: 0xdevalias; Comment Subject: Re: [OJ/gobuster] Added custom prefix, suffix functionalities and -random-agent. (#56) GoGet must now then return one additional argument too: func GoGet(s *State, url, uri, cookie string) (*int, *int64, error) { return MakeRequest(s, url+uri, cookie) } Would you like to keep sending this error up to the moon? Before: wildcardResp, _ := GoGet(s, s.Url, fmt.Sprintf("%s", guid), s.Cookies) After: wildcardResp, _, err := GoGet(s, s.Url, fmt.Sprintf("%s", guid), s.Cookies) And same in all cases where GoGet is being called? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#56 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAuA4241hpjHsxZVqErvaqgjIojENK9Oks5tUrL3gaJpZM4PcM2Q>.

BBerastegui · 2018-02-14T12:55:46Z

I'll left TODO's for error handling as you guys prefer to do it.

0xdevalias · 2018-02-16T23:31:49Z

Thanks for making those changes @BBerastegui !

@OJ I haven't actually run this PR to confirm it all works as expected, but assuming it does, i'm pretty happy with how it looks from a code perspective. So if it works/you're happy, LGTM!

0xdevalias · 2018-02-16T23:33:43Z

libgobuster/dir.go

@@ -59,6 +61,22 @@ func MakeRequest(s *State, fullUrl, cookie string) (*int, *int64) {
 		req.Header.Set("User-Agent", s.UserAgent)
 	}

+	if s.RandomUserAgent && s.UserAgent == "" {
+		b, err := ioutil.ReadFile(s.UserAgentsFile)


We could probably make this more efficient by reading in the UserAgents file once at the beginning and caching it, rather than on every request. The current way might slow things down a bunch, particularly if there are a lot of agents in the file.

That's true. Where should I put it then?

I thought in SetupDir(), by adding UserAgents []string to the State and using it to store the array of User-Agents.

But in this case, what do we do with the error? Panic? (as I put in the example), and also, this can't be done there (in SetupDir):

s.Printer = PrintDirResult s.Processor = ProcessDirEntry s.Setup = SetupDir

As the ProcessDirEntry is run first in statehelpers.go and it already requires the User-Agent to be set.

I think some of the checks belong in ValidateState() or ValidateDirModeState() if they aren't already: https://github.com/OJ/gobuster/blob/master/libgobuster/statehelpers.go

Adding UserAgents []string to the state would be the correct choice I think.

The 'easy' choice is to panic here, the 'better' choice is probably to add support for an 'error' return value to SetupDir as well. Which would also require similar to be done for SetupDns. And then handled somehow wherever they are called from:

Added to the definition of SetupFunc here: https://github.com/OJ/gobuster/blob/master/libgobuster/state.go#L24

The error handling changed from checking a bool to checking for error here. Not actually sure what the bool is representing here.. we may need both it and the error..? But I think it's just meant to be a 'rudimentary' error check: https://github.com/OJ/gobuster/blob/master/libgobuster/state.go#L80

There is a slight chance there are other changes required, these were the only ones I saw when looking just now though.

Bonus points if you then add an error return to Process: https://github.com/OJ/gobuster/blob/master/libgobuster/state.go#L76

And the associated error check for that at https://github.com/OJ/gobuster/blob/master/main.go#L76

Is ProcessDirEntry actually run in statehelpers? The only line I see is https://github.com/OJ/gobuster/blob/master/libgobuster/statehelpers.go#L53 which I believe is just storing the pointer to the function. It's called elsewhere. So I think this would be fine?

0xdevalias · 2018-02-16T23:36:59Z

README.md

 * `-r` - follow redirects.
 * `-s <status codes>` - comma-separated set of the list of status codes to be deemed a "positive" (default: `200,204,301,302,307`).
+* `-suffix <suffix>` - append a custom suffix for directory brute forces.


Can you add the RandomUserAgent/UserAgentFile stuff to the README params as well?

0xdevalias · 2018-02-16T23:38:39Z

libgobuster/dir.go

+		lines := strings.Split(string(b), "\n")
+		var uas []string
+		for _, line := range lines {
+			if !strings.HasPrefix(line, "#") && len(line) > 0 {


Not 100%, but you may need to trim pre/post whitespace on here, otherwise a blank line with a few spaces might be counted as a valid UA.

OJ · 2018-02-19T01:13:26Z

Thanks for the efforts you're both putting into this 👍

BBerastegui · 2018-03-12T10:43:13Z

libgobuster/dir.go

@@ -159,7 +148,7 @@ func ProcessDirEntry(s *State, word string, resultChan chan<- Result) {
 	// TODO: Error propagation handling
 	dirResp, dirSize, err := GoGet(s, s.Url, prefix+word+suffix, s.Cookies)
 	if err != nil {
-		panic(err)
+		//panic(err)


I made this terrible thing here because I realized that the 302 redirect was causing a panic here.

Which '302 redirect'? Can you give a specific example/site it happens on for debugging.

Definitely will need to actually do something with the error here before it can be merged, otherwise it will fail silently, and that's bad.

Without running it/seeing it in practice I can't really give a good idea of how it should be handled. If it's returning an error where it isn't really an error, then we can probably catch that case and handle it somewhere appropriately.

Funny thing is that now, when trying to reproduce it, I realised that it's happening in the host I was testing it, for a 302 redirect:

Gobuster v1.4.1 OJ Reeves (@TheColonial) ===================================================== ===================================================== [+] Mode : dir [+] Url/Domain : https://king.com/ [+] Threads : 10 [+] Wordlist : /dev/fd/63 [+] Status codes : 200,204,301,302,307 ===================================================== panic: Redirect code: 302 goroutine 15 [running]: panic(0x22a900, 0xc4203d1ea8) /usr/local/go/src/runtime/panic.go:500 +0x1a1 github.com/bberastegui/gobuster/libgobuster.ProcessDirEntry(0xc4200ce000, 0xc4203dc16a, 0x6, 0xc4203109c0) /[REDACTED]/go/src/github.com/bberastegui/gobuster/libgobuster/dir.go:151 +0x438 github.com/bberastegui/gobuster/libgobuster.Process.func1(0xc420310960, 0xc4200ce000, 0xc4203109c0, 0xc4203dc1c0) /[REDACTED]/go/src/github.com/bberastegui/gobuster/libgobuster/state.go:116 +0x8f created by github.com/bberastegui/gobuster/libgobuster.Process /[REDACTED]/go/src/github.com/bberastegui/gobuster/libgobuster/state.go:122 +0x1a9``` But not on other redirects like 301, for example.

0xdevalias · 2018-03-12T20:49:25Z

libgobuster/statehelpers.go

+	if errorList.ErrorOrNil() == nil && s.RandomUserAgent {
+		b, err := ioutil.ReadFile(s.UserAgentsFile)
+		if err != nil {
+			errorList = multierror.Append(errorList, fmt.Errorf("[!] Error when reading the User-Agents file"))


We're masking the cause of the error here by not including it. Could just add the error directly to the errorList, though then you can't add your additional context. At the very least I would include the error message with something like:

fmt.Errorf("[!] Error when reading the User-Agents file: %s", err)

Even that isn't super ideal. Could potentially use %+v to get more context in it? In my own code, I normally would wrap it with another library:

errors.Wrap(err, "[!] Error when reading the User-Agents file")

That would be my best case scenario to do things 'right', but would require adding in another dependency (which personally I don't see as a huge issue).

I like the %+v:

Gobuster v1.4.1 OJ Reeves (@TheColonial) ===================================================== 1 error occurred: * [!] Error when reading the User-Agents file: open user-agents.txt: no such file or directory```

0xdevalias · 2018-03-12T20:54:05Z

libgobuster/statehelpers.go

+		lines := strings.Split(string(b), "\n")
+		var uas []string
+		for _, line := range lines {
+			if !strings.HasPrefix(line, "#") && len(line) > 0 {


I would probably call strings.TrimSpace on the line before checking for the prefix:

https://golang.org/pkg/strings/#TrimSpace

0xdevalias · 2018-03-12T21:00:52Z

libgobuster/statehelpers.go

@@ -170,6 +171,22 @@ func ValidateDirModeState(
 		}
 	}

+	// Read the user agents file
+	if errorList.ErrorOrNil() == nil && s.RandomUserAgent {
+		b, err := ioutil.ReadFile(s.UserAgentsFile)


ioutil.ReadFile will read the entire file into memory, then you split on newlines, remove comments/etc. It may not matter super much here, but there is a potential that we read in a lot of stuff that is never required, etc.

Following the pattern used for reading the , we could use bufio.NewScanner to read/process the file line by line:

https://github.com/OJ/gobuster/blob/master/libgobuster/state.go#L143

https://github.com/OJ/gobuster/blob/master/libgobuster/state.go#L158-L168

So an implementation in this case might look something like:

for scanner.Scan() { if s.Terminate { break } line := strings.TrimSpace(scanner.Text()) // Skip "comment" (starts with #), as well as empty lines if !strings.HasPrefix(word, "#") && len(line) > 0 { uas = append(uas, line) } }

If we wanted to be fancy, we could probably refactor the common functionality between them into a helper function.. But that is probably beyond what is required here.

OJ · 2018-08-27T02:24:59Z

Apologies for the delay on this, can I please request that you rebase your changes on 2.0.0 now that I've finally landed #79? Thank you so much.

Shaked · 2019-03-02T11:48:45Z

Hey @BBerastegui do you need help with this pull request? Seems very useful!

BBerastegui · 2019-03-06T15:04:36Z

Hey @Shaked, to be completely honest, I think my knowledge of Go is pretty limited and I think I can't catch up with this now.

Feel free to get my commit and rebase the changes agains the latest version if you want :)

OJ · 2019-03-22T00:00:32Z

I'll see if I can put some time into this later. Sorry for the epic divergence. Dat Open Source Lyfe yo!

firefart · 2019-11-27T14:00:39Z

This should all be already implemented on the v3.1-working branch

Added custom prefix and suffix functionalities.

7b06faa

0xdevalias mentioned this pull request Oct 7, 2017

Add 'chomp slash' feature #60

Open

0xdevalias reviewed Oct 8, 2017

View reviewed changes

Added -random-agent feature

d5194e8

BBerastegui changed the title ~~Added custom prefix and suffix functionalities.~~ Added custom prefix, suffix functionalities and -random-agent. Oct 11, 2017

Forgot to actually add the User-Agent to the request

55d7eec

What a terrible mess.

e91f5fd

BBerastegui and others added 3 commits January 17, 2018 00:10

More fixes into the disaster

0471e81

Now I think we are talking

a29e4fb

Merge branch 'master' into master

c04afd1

0xdevalias reviewed Jan 17, 2018

View reviewed changes

Added suggestions and formatting.

b8a781b

Left TODO of error handling.

226f30d

BBerastegui mentioned this pull request Feb 14, 2018

Add ability to bust with placeholders. #71

Closed

More fixes and adaptation

e85d0b1

0xdevalias reviewed Feb 16, 2018

View reviewed changes

BBerastegui added 2 commits February 22, 2018 13:10

Added to the README the -random-agent info

dbe647e

The user-agents file is read only once now

7be81df

BBerastegui commented Mar 12, 2018

View reviewed changes

0xdevalias suggested changes Mar 12, 2018

View reviewed changes

0xdevalias mentioned this pull request Mar 12, 2018

Arbitrary headers, POST support, POST body support, Content-Type support, linting #76

Closed

Removed spaces from the UA files, error formatting

265d03f

OJ added this to the v3.1 milestone Oct 1, 2019

Derrankyle approved these changes Nov 27, 2019

View reviewed changes

firefart closed this Jun 20, 2020

Added custom prefix, suffix functionalities and -random-agent. #56

Added custom prefix, suffix functionalities and -random-agent. #56

Conversation

BBerastegui commented Sep 19, 2017 • edited

0xdevalias left a comment

Choose a reason for hiding this comment

BBerastegui commented Oct 10, 2017 • edited

BBerastegui commented Oct 10, 2017

OJ commented Jan 14, 2018

BBerastegui commented Jan 16, 2018

OJ commented Jan 16, 2018

BBerastegui commented Jan 16, 2018

BBerastegui commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xdevalias commented Feb 14, 2018 via email

BBerastegui commented Feb 14, 2018

0xdevalias commented Feb 14, 2018 via email

BBerastegui commented Feb 14, 2018

0xdevalias commented Feb 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xdevalias Feb 16, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OJ commented Feb 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OJ commented Aug 27, 2018

Shaked commented Mar 2, 2019

BBerastegui commented Mar 6, 2019

OJ commented Mar 22, 2019

firefart commented Nov 27, 2019

BBerastegui commented Sep 19, 2017 •

edited

BBerastegui commented Oct 10, 2017 •

edited

0xdevalias Feb 16, 2018 •

edited