New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default URL filter: exclude localhost and private address spaces #543

merged 1 commit into from Mar 7, 2018


None yet
2 participants

sebastian-nagel commented Mar 7, 2018

To avoid that information is leaked and exposed in a public search interface, a web crawler should never follow links to localhost, loop-back addresses and private address spaces, see commoncrawl/news-crawl#21. These are excluded by regular expressions for http/https URLs.

Of course, could comment out the rules to enable test crawls of the local Apache web server.

@jnioche jnioche merged commit 62c6e3d into DigitalPebble:master Mar 7, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed

@jnioche jnioche added the enhancement label Mar 7, 2018

@jnioche jnioche added this to the 1.8 milestone Mar 7, 2018


This comment has been minimized.


jnioche commented Mar 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment