• A little like that j-thing, only in Go.

    Go 4,118 385 Updated May 20, 2017
  • A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

    Go 551 76 Updated Apr 8, 2017
  • Proper URL escaping as per RFC3986

    Go 4 4 Updated Mar 24, 2017
  • Package rehttp implements a Go HTTP transport that handles retries.

    Go 131 3 Updated Mar 24, 2017
  • Polite, slim and concurrent web crawler.

    Go 1,327 153 Updated Mar 24, 2017
  • tiny Go library to normalize URLs

    Go 256 36 Updated Mar 24, 2017