Comparing simple Go and Python web crawler
Go Python Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
LICENSE
README.md
crawl.go
crawl.py
run.sh

README.md

GopherSnakeCrawlers

Comparing simple Go and Python web crawler

For each crawler, the first argument is the source link, the second is the number of workers and the third is the number of pages to fetch.

For example: python crawl.py http://google.com 1 10

run.sh runs the crawler multiple times with different worker, page fetch parameters. The first argument is to specify go or python and the second argument is the source link. For go, it expects an executable with the name crawl be present in the current directory.

For example: ./run.sh python http://google.com

More details at: http://venkat.io/posts/concurrent-crawling/