This repository is private.
All pages are served over SSL and all pushing and pulling is done over SSH.
No one may fork, clone, or view it unless they are added as a member.
Every repository with this icon (
) is private.
Every repository with this icon (
) is private.
This repository is public.
Anyone may fork, clone, or view it.
Every repository with this icon () is public.
Every repository with this icon (
) is public.
commit ae1b43cfd624364af1065cc5b8048634da5f19dd
tree 92ae9f02e6efc8b17276922c1b0352232b335562
parent e475a64e6bbf4e2c7a0bcc5e6407dfa7881b9ad3
tree 92ae9f02e6efc8b17276922c1b0352232b335562
parent e475a64e6bbf4e2c7a0bcc5e6407dfa7881b9ad3
anemone / README.rdoc
Anemone
Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
See anemone.rubyforge.org for more information.
Features
- Multi-threaded design for high performance
- Tracks 301 HTTP redirects to understand a page’s aliases
- Built-in BFS algorithm for determining page depth
- Allows exclusion of URLs based on regular expressions
- Choose the links to follow on each page with focus_crawl()
- HTTPS support
- Records response time for each page
- CLI program can list all pages in a domain, calculate page depths, and more
Examples
See the scripts under the lib/anemone/cli directory for examples of several useful Anemone tasks.
Requirements
- nokogiri
- robots
