An attempt at a python web crawler
Create a mysql table. If you don't know how to do this I recommend using phpmyadmin. Collation should be
settings.py with your database settings.
Now you can add links to the
link table directly or you can edit the
SEED_LINKS list in
settings.py and run
python explorer.py --seed
If you're in a hurry you can leave
SEED_LINKS alone and just seed from the command line with
python explorer.py -seed http://myfirstseed.com http://mysecondseed.net ...
The seeding procedure is only required if you have no uncrawled links in your database (ie: first time running the crawler).
After getting a few pages, start a
You can run as many explorers and harvesters concurrently as you like.
How it works
Each explorer gets uncrawled links from the
link table, retrieves their html contents and dumps them into the
The harvester gets an unharvested html content from the
webpage table, extracts links, and stores them in as uncrawled links in the
- Automate explorer/harvester relation
- Better error handling