Building block for web crawler in Go #golang. (Not yet ready) net/http/client enhanced with timeouts, limits and robots.txt support.
Fetching latest commit…
Cannot retrieve the latest commit at this time
What ==== HTTP client library, command line worker for Heroshi, web crawler at https://github.com/temoto/heroshi This project has served its purposes and is now abandoned.