Useless fork of meg for just ingesting a url list instead of a hosts + paths lists. Created this real quick in order to get output of two approaches to match for easy grepping. Most likely will not update this repo as the feature should be committed upstream eventually.
See the original repo for more details: meg
$ go get -u github.com/tomnomnom/meg
$ go build .
Request many paths for many hosts
Usage:
meg [urlsFile] [outputDir]
Options:
-b, --body <val> Set the request body
-c, --concurrency <val> Set the concurrency level (default: 20)
-d, --delay <millis> Milliseconds between requests to the same host (default: 5000)
-H, --header <header> Send a custom HTTP header
-L, --location Follow redirects / location header
-r, --rawhttp Use the rawhttp library for requests (experimental)
-s, --savestatus <status> Save only responses with specific status code
-t, --timeout <millis> Set the HTTP timeout (default: 10000)
-v, --verbose Verbose mode
-X, --method <method> HTTP method (default: GET)
Defaults:
urlsFile: ./urls
outputDir: ./out
URLS file format:
http://acme.wut/robots.txt
Examples:
meg urls.txt output