Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
spiderfetch
.gitignore
.travis.yml
LICENSE
README.rst
dev-requirements.txt
setup.cfg
setup.py
tox.ini

README.rst

spiderfetch

https://badge.fury.io/py/spiderfetch.png https://travis-ci.org/numerodix/spiderfetch.png?branch=master

Installation

$ pip install spiderfetch
$ spiderfetch

Usage

Fetching

Fetch all urls matching 2008.*.ogg from a page:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg

To dump the urls to a file instead of fetching:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg --dump > urls

Spidering

Spider a site to depth 3 while pausing 2 seconds between fetches. The urls that will be considered when spidering must match .*:

$ spiderfetch --host http://en.wikipedia.org --depth 3 --pause 2 '.*'