Support the Robots meta tag

[sylvinus]

http://www.robotstxt.org/meta.html

Not sure if Common Crawl already filters those pages, but we should do it on our side too anyway.

Some pointers:

• Add a is_indexable method to Document
• HTMLDocument.is_indexable() should test for the presence of noindex in self.head_metas["robots"]
• Documents should be skipped accordingly in indexer.py
• To support nofollow we could simply test for its presence in close_tag

[sylvinus]

We should also support the x-robots-tag http header:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#using-the-robots-meta-tag

[sylvinus]

@jhildreth started a branch with some good work that should be merged:
master...jhildreth:feature/robots-tag