This repository has been archived by the owner on Mar 7, 2021. It is now read-only.

simplecrawler 1.1.6

fredrikekelund released this 06 Oct 12:14

· 77 commits to master since this release

New features

Sitemap directives in /robots.txt are now added to the queue if Crawler#respectRobotsTxt is truthy.

Bug fixes

#398 - fix issue where multiple cookies weren't properly serialized for outbound requests
#400 - fix issue where <meta name="robots"> tags weren't properly parsed

Assets 2