C# web crawler build for speed and flexibility.
Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to take complete control over the crawl process.
Abot targets .NET version 4.0.
Latest documentation: http://code.google.com/p/abot/
Community Forum: http://groups.google.com/group/abot-web-crawler