This repository has been archived by the owner on Jun 26, 2020. It is now read-only.

Support for robots.txt #6

Open

missinglink opened this issue Sep 22, 2013 · 0 comments

Owner

missinglink commented Sep 22, 2013

Obey robots.txt. Minimum functonality:

cancel all requests which globally disallow the huntsman User-Agent

User-agent: huntsman
Disallow: /

cancel all requests for urls which match Disallow statements

User-agent: huntsman
Disallow: /private/area

The text was updated successfully, but these errors were encountered:

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.