Simple pure-python spider trap for testing crawlers
Python
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
README.rst
setup.py
spidertrap
spidertrap.py

README.rst

SpiderTrap

An http server that responds with an ever-expanding web of links. Use it to test, benchmark, and confuse web crawlers.

Implemented as a simple python script with no dependencies other than the standard library. Works with python 2.7 and python 3.

Installation

$ pip install spidertrap

Usage

spidertrap [-h] [-p PORT] [--fanout FANOUT]

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  [9999]
  --fanout FANOUT       Number of links to create on each page [2]

Licence

MIT