Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Anemone web-spider framework
branch: master

This branch is 24 commits ahead, 147 commits behind chriskite:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
bin
lib
spec
CHANGELOG.md
LICENSE.txt
README.rdoc
anemone.gemspec

README.rdoc

Anemone

Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.

Features:

  • Multi-threaded design for high performance

  • Tracks 301 HTTP redirects to understand a page's aliases

  • Built-in BFS algorithm for determining page depth

  • Allows exclusion of URLs based on regular expressions

Examples

See the scripts under lib/anemone/cli directory for examples of several useful Anemone tasks.

REQUIREMENTS

  • nokogiri

Something went wrong with that request. Please try again.