getallurls (gau)

getallurls (gau) fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, Common Crawl, and URLScan for any given domain. Inspired by Tomnomnom's waybackurls.

Resources

• Usage
• Installation
• ohmyzsh note

Usage:

Examples:

$ printf example.com | gau
$ cat domains.txt | gau --threads 5
$ gau example.com google.com
$ gau --o example-urls.txt example.com
$ gau --blacklist png,jpg,gif example.com

To display the help for the tool use the -h flag:

$ gau -h

Flag	Description	Example
--blacklist	list of extensions to skip	gau --blacklist ttf,woff,svg,png
--fc	list of status codes to filter	gau --fc 404,302
--from	fetch urls from date (format: YYYYMM)	gau --from 202101
--ft	list of mime-types to filter	gau --ft text/plain
--fp	remove different parameters of the same endpoint	gau --fp
--json	output as json	gau --json
--mc	list of status codes to match	gau --mc 200,500
--mt	list of mime-types to match	gau --mt text/html,application/json
--o	filename to write results to	gau --o out.txt
--providers	list of providers to use (wayback,commoncrawl,otx,urlscan)	gau --providers wayback
--proxy	http proxy to use (socks5:// or http://	gau --proxy http://proxy.example.com:8080
--retries	retries for HTTP client	gau --retries 10
--timeout	timeout (in seconds) for HTTP client	gau --timeout 60
--subs	include subdomains of target domain	gau example.com --subs
--threads	number of workers to spawn	gau example.com --threads
--to	fetch urls to date (format: YYYYMM)	gau example.com --to 202101
--verbose	show verbose output	gau --verbose example.com
--version	show gau version	gau --version

Configuration Files

gau automatically looks for a configuration file at $HOME/.gau.toml or%USERPROFILE%\.gau.toml. You can specify options and they will be used for every subsequent run of gau. Any options provided via command line flags will override options set in the configuration file.

An example configuration file can be found here

Installation:

From source:

$ go install github.com/lc/gau/v2/cmd/gau@latest

From github :

git clone https://github.com/lc/gau.git; \
cd gau/cmd; \
go build; \
sudo mv gau /usr/local/bin/; \
gau --version;

From binary:

You can download the pre-built binaries from the releases page and then move them into your $PATH.

$ tar xvf gau_2.0.6_linux_amd64.tar.gz
$ mv gau /usr/bin/gau

From Docker:

You can run gau via docker like so:

docker run --rm sxcurity/gau:latest --help

You can also build a docker image with the following command

docker build -t gau .

and then run it

docker run gau example.com

Bear in mind that piping command (echo "example.com" | gau) will not work with the docker container

ohmyzsh note:

ohmyzsh's git plugin has an alias which maps gau to the git add --update command. This is problematic, causing a binary conflict between this tool "gau" and the zsh plugin alias "gau" (git add --update). There is currently a few workarounds which can be found in this Github issue.

Useful?

Donate to CommonCrawl
Donate to the InternetArchive