Flag to ignore errors and keep running (or -sr bug fix)

[intrd]

Hello guys, could you add a flag to ignore errors like this and keep running httpx?

http://www.target.com:80/xxx
https://target.com/xx.xx
https://www.target.com/xxxxxx/?hl=nl/
http://www.target.com:80/xxxx
[FTL] Could not write response, at path 'output/target.com:80_%22:269092299880021,%224456808911257%22:117599551732038,%224456808791254%22:219099868222532,%224456808631250%22:117241395101609,%224456808471246%22:379290365485680,%224456808271241%22:217348818399031,%224456808071236%22:512726018751670,%224456807991234%22:291913630927125,%224456807831230%22:291001267685617,%224456807791229%22:378181278930281%7D.txt', to disc.

...and httpx quit.

This is a problem when we are processing a giant url listing and at some point due to illegal character or very long url it cannot write to disk.

thanks!
awesome tool.

[ehsandeep]

Hey @intrd,

Can you add some sample URLs where it gets failed, so we can make changes accordingly.

[intrd]

Hey @intrd,

Can you add some sample URLs where it gets failed, so we can make changes accordingly.

sure, this input breaks the httpx:

http://target.com:80/%22:269092299880021,%224456808911257%22:117599551732038,%224456808791254%22:219099868222532,%224456808631250%22:117241395101609,%224456808471246%22:379290365485680,%224456808271241%22:217348818399031,%224456808071236%22:512726018751670,%224456807991234%22:291913630927125,%224456807831230%22:291001267685617,%224456807791229%22:378181278930281%7D

maybe a good idea is to hash or randomize the url path and stores like this:

/target.com_80/32e152b6824592d175fc8cfda4237433e595292770a67c543c4992d30cb0f9f2

[intrd]

Dirty/temporary fix @ httpx/cmd/httpx/httpx.go:

"encoding/hex"
"crypto/md5"

...

        // store responses in directory
        if scanopts.StoreResponse {
                splitted := strings.Split(domain, "/")
                path := ""
                if len(splitted) == 1 {
                        path = "/"
                }else{
                        path = splitted[1]
                }
                domain := splitted[0]
                hasher := md5.New()
                hasher.Write([]byte(path))
                out := hex.EncodeToString(hasher.Sum(nil))
                fullpath := "output/"+domain+"/"+out
                if _, err := os.Stat("output/"+domain); os.IsNotExist(err) { 
                        os.MkdirAll("output/"+domain, 0700)
                }
                err := ioutil.WriteFile(fullpath, []byte(resp.Raw), 0644)
                if err != nil {
                        gologger.Fatalf("Could not write response, at path to disc: %s", fullpath)
                }
        }

$ find output/

output/
output/target1.com
output/target1.com/a8f5f167f44f4964e6c998dee827110c
output/target2.com
output/target2.com/a660a4c512ce611d365624fe1bc6b5c6
output/target2.com/22bd4dae3f7b3f28e87813a4ccd13a12
output/help.target3.com
output/help.target3.com/2f8a6bf31f3bd67bd2d9720c58b19c9a

[ehsandeep]

@intrd This is fixed with #113