Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

19 lines (14 sloc) 0.572 kb

Anemone

DESCRIPTION

Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.

FEATURES

  • Multi-threaded design for high performance

  • Tracks 301 HTTP redirects to understand a page's aliases

  • Built-in BFS algorithm for determining page depth

  • Allows exclusion of URLs based on regular expressions

REQUIREMENTS

  • nokogiri

EXAMPLES

See the bin directory for several examples of useful Anemone tasks.

Jump to Line
Something went wrong with that request. Please try again.