WaybackUnifier allows you to take a look at how a file has ever looked by aggregating all versions of this file, and creating a unified version that contains every line that has ever been in it.
Go is required.
go get github.com/mhmdiaa/waybackunifier
This will download the code, compile it, and leave a
waybackunifier binary in $GOPATH/bin.
-concurrency int Number of requests to make in parallel (default 1) -output string File to save results in (default "output.txt") -sub string list of comma-separated substrings to look for in snapshots (snapshots will only be considered if they contnain one of them) (default "Disallow,disallow") -url string URL to unify versions of (without protocol prefix) (default "site.com/robots.txt")
The settings are by default suitable for unifying robots.txt files. Feel free to change the value of
-sub to anything else, or supply an empty string to get all versions of a file without filtering.
Note: Lines are saved unordered for performance reasons