Skip to content

rehberg/crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

"My Web Crawler"

That web crawler thing I always wanted to do...

Work in progress....

Using Python and Perl to build a Web crawler that simply goes out and fetches things randomly. It is (or at least will be) robots.txt aware.

I don't have a real goal in mind at the moment, but perhaps it will have a use other than whetting my own problem-solving appetite.

About

Web Crawler scripts in Python and Perl

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors