crawler

To start the crawler, execute the following command from the shell prompt:

python crawler.py

This will write all links to a file called 'all_links'

Note: You would need to add links-finder/link-finder directory to your python path