Blacklist URLs

[iansinnott]

You may not want every URL to be indexed. There should be a way of blacklisting URLs.

Thoughts:

• simple string match: example.com doesn't index anything that starts with that URL (if we make the https?:// optional)
• regex blacklist, when you want to get really specific
• file-based blacklist: point bp to a file on disk and we'll read the blacklist from there. no need to even store the blacklisted URL patterns in bp

[nfcampos]

I think would also be good to have some sensible defaults? eg.

• don't index anything starting with chrome-extension://
• or maybe even better, only index URLs starting with http or https
• don't index localhost
• maybe don't index any bare IPs, only domains

[iansinnott]

Good points. There are a few of these hardcoded at present but they should be configurable. things like about:blank or chrome://* are currently excluded if i remember correctly.