[Feature Request] Switch to enable/disable urls from subdomains of base domain

[Rhynorater]

Hey Tom!

Great tool! I was wondering if it would be possible for you to add a command-line arg to select URLs only from the base domain provided. The new WayBack Machine query would look like this:

http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&collapse=urlkey

as opposed to the one that pulls all subdomains too (currently in your code) which is this:

http://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&collapse=urlkey

I propose that this param should be called "with_subs" and should be Boolean defaulting to True.

Thanks,
Justin

[Rhynorater]

Also, note, I did try to implement this, but Go doesn't like me...

I tried to add a CMD param like so:

var dates bool                                                                                                                                                                                            
var withSubs bool                                                                                                                                                                                         
flag.BoolVar(&dates, "dates", false, "show date of fetch in the first column")                                                                                                                            
flag.BoolVar(&withSubs, "withSubs", true, "pull URLs for subdomains of the hosts provided as well")

and then pass it into the functions getWaybackURLs and getCommonCrawlURLs via this:

resp, err := fetch(domain, withSubs)

Once I modified the resulting functions to look like this:

func getWaybackURLs(domain string, withSubs bool) ([]wurl, error) {
        if withSubs {
                res, err := http.Get(
                        fmt.Sprintf("http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&collapse=urlkey", domain),
                )
        } else {
                res, err := http.Get(
                        fmt.Sprintf("http://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&collapse=urlkey", domain),
                )
        }

But Go didn't like that... I don't speak Go hahaha.

[tomnomnom]

Hey, @Rhynorater! Good idea :)

I'll have a poke at it now

[tomnomnom]

I've added a --no-subs option in 41bb26a

The only real problem with it is that the searches still often include subdomains, so the actual behaviour doesn't quite line up with the expected behaviour.

I'll add some additional filtering I think

[tomnomnom]

@Rhynorater I've added some output filtering so you only get results for the exact input domain if you specify the --no-subs option in 58bbafe

Give it a try and let me know if you hit any problems :)

[Rhynorater]

Thanks Tom! Sounds great!

…

On Tue, Aug 28, 2018 at 11:53 AM, Tom Hudson ***@***.***> wrote: I've added a --no-subs option in 41bb26a <41bb26a> The only real problem with it is that the searches still often include subdomains, so the actual behaviour doesn't *quite* line up with the expected behaviour. I'll add some additional filtering I think — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC2_r0a33rDEPuQeWmSFHeicYg0MYgs9ks5uVWdwgaJpZM4WPnr3> .

-- Justin Gardner Cell: (804)525-8089