chriskite / anemone
watch download tarball
public
Forks21Right
Watchers232Right
Description: Anemone web-spider framework
Homepage: http://anemone.rubyforge.org
Clone URL: git://github.com/chriskite/anemone.git
name age
history
message
file CHANGELOG.rdoc Sun Nov 01 15:18:59 -0800 2009 updated README and CHANGELOG [chriskite]
file LICENSE.txt Tue Apr 14 12:14:47 -0700 2009 initial import [Chris Kite]
file README.rdoc Sun Nov 01 15:18:59 -0800 2009 updated README and CHANGELOG [chriskite]
file anemone.gemspec Sun Nov 01 14:58:12 -0800 2009 use per-crawl options instead of specifying opt... [chriskite]
directory bin/ Thu Oct 22 20:51:37 -0700 2009 change interperter line on anemoen cli bin [chriskite]
directory lib/ Sun Nov 01 15:18:59 -0800 2009 updated README and CHANGELOG [chriskite]
directory spec/ Sun Nov 01 15:12:05 -0800 2009 refactor Core#skip_links_like [chriskite]
README.rdoc

Anemone

Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.

See anemone.rubyforge.org for more information.

Features

  • Multi-threaded design for high performance
  • Tracks 301 HTTP redirects to understand a page’s aliases
  • Built-in BFS algorithm for determining page depth
  • Allows exclusion of URLs based on regular expressions
  • Choose the links to follow on each page with focus_crawl()
  • HTTPS support
  • Records response time for each page
  • CLI program can list all pages in a domain, calculate page depths, and more

Examples

See the scripts under the lib/anemone/cli directory for examples of several useful Anemone tasks.

Requirements

  • nokogiri
  • robots