Building block for web crawler in Go #golang. (Not yet ready) net/http/client enhanced with timeouts, limits and robots.txt support.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
heroshi
limitmap
slow-server
README
worker.go
worker_main.go

README

What
====

HTTP client library, command line worker for Heroshi, web crawler at https://github.com/temoto/heroshi

This project has served its purposes and is now abandoned.