Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

GopherSnakeCrawlers

Comparing simple Go and Python web crawler

For each crawler, the first argument is the source link, the second is the number of workers and the third is the number of pages to fetch.

For example: python crawl.py http://google.com 1 10

run.sh runs the crawler multiple times with different worker, page fetch parameters. The first argument is to specify go or python and the second argument is the source link. For go, it expects an executable with the name crawl be present in the current directory.

For example: ./run.sh python http://google.com

More details at: http://venkat.io/posts/concurrent-crawling/

About

Comparing simple Go and Python web crawler

Resources

License

Releases

No releases published
You can’t perform that action at this time.