Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

spiderfetch

https://badge.fury.io/py/spiderfetch.png https://travis-ci.org/numerodix/spiderfetch.png?branch=master

Installation

$ pip install spiderfetch
$ spiderfetch

Usage

Fetching

Fetch all urls matching 2008.*.ogg from a page:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg

To dump the urls to a file instead of fetching:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg --dump > urls

Spidering

Spider a site to depth 3 while pausing 2 seconds between fetches. The urls that will be considered when spidering must match .*:

$ spiderfetch --host http://en.wikipedia.org --depth 3 --pause 2 '.*'

About

A modular web spider

Resources

License

Packages

No packages published

Languages