This repository is private.
All pages are served over SSL and all pushing and pulling is done over SSH.
No one may fork, clone, or view it unless they are added as a member.
Every repository with this icon (
) is private.
Every repository with this icon (
) is private.
This repository is public.
Anyone may fork, clone, or view it.
Every repository with this icon () is public.
Every repository with this icon (
) is public.
commit 7b58fecdb142bc126967ba70888ad9c61eae193a
tree 625253b63efb93691798f84cb26c6986b91d65e4
parent 6bbe30cd489b385fdb83113e79c3af9a803e4279
tree 625253b63efb93691798f84cb26c6986b91d65e4
parent 6bbe30cd489b385fdb83113e79c3af9a803e4279
anemone /
| name | age | message | |
|---|---|---|---|
 |
CHANGELOG.rdoc | Wed Dec 16 12:06:09 -0800 2009 | |
| |
LICENSE.txt | Tue Apr 14 12:14:47 -0700 2009 | |
| |
README.rdoc | Wed Dec 16 12:06:09 -0800 2009 | |
| |
anemone.gemspec | Wed Nov 04 09:06:58 -0800 2009 | |
 |
bin/ | Thu Oct 22 20:51:37 -0700 2009 | |
| |
lib/ | Wed Dec 16 12:05:50 -0800 2009 | |
| |
spec/ | Wed Dec 16 12:05:50 -0800 2009 |
README.rdoc
Anemone
Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
See anemone.rubyforge.org for more information.
Features
- Multi-threaded design for high performance
- Tracks 301 HTTP redirects to understand a page’s aliases
- Built-in BFS algorithm for determining page depth
- Allows exclusion of URLs based on regular expressions
- Choose the links to follow on each page with focus_crawl()
- HTTPS support
- Records response time for each page
- CLI program can list all pages in a domain, calculate page depths, and more
- Obey robots.txt
- In-memory or persistent storage of pages during crawl, using TokyoCabinet or PStore
Examples
See the scripts under the lib/anemone/cli directory for examples of several useful Anemone tasks.
Requirements
- nokogiri
- robots
