Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
Multi-threaded design for high performance
Tracks 301 HTTP redirects to understand a page's aliases
Built-in BFS algorithm for determining page depth
Allows exclusion of URLs based on regular expressions
See the scripts under lib/anemone/cli directory for examples of several useful Anemone tasks.