A little like that j-thing, only in Go.
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Proper URL escaping as per RFC3986
Package rehttp implements a Go HTTP transport that handles retries.
Polite, slim and concurrent web crawler.
tiny Go library to normalize URLs