This repository has been archived by the owner on Mar 7, 2021. It is now read-only.

simplecrawler 1.0.3

fredrikekelund released this 22 Oct 10:44

· 146 commits to master since this release

Bug fixes

#330 - fixed an issue that would make the crawler throw an error if it was started, then stopped, then started again.
#328 - fixed an issue where multiple directives in <meta name="robots"> tags would make simplecrawler not recognize the nofollow directive

Assets 2